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Preface 


This  symposium  provides  an  opportunity  to  present  and  discuss  Alberta's  Educational 
Quality  Indicators  (EQI)  initiative.  At  the  1989  meeting  of  the  Canadian  Educational 
Researchers'  Association,  McEwen  and  Zatko  presented  the  conceptual  framework 
and  implementation  plan  for  this  initiative.  This  symposium  brings  together  five  of  the 
project  teams  to  provide  their  perspective  on  the  initiative  and  to  report  their  progress 
to  date  in  developing  local  indicator  systems  which  have  the  involvement  and  support 
of  their  respective  communities. 

Alberta  Education  is  collaborating  with  twelve  school  jurisdictions  to  develop  indicator 
systems  to  measure  the  success  of  the  educational  enterprise  in  the  province.  The 
indicators  will  provide  information  to  assist  in  assessing  the  quality  of  educational 
programs  and  the  delivery  system  by  focusing  on  student  outcomes.  The  proposed 
system  has  taken  into  consideration  and  reflects  government  policy  and  the  goals  of 
schooling.  It  addresses  two  essential  questions: 

1 . Are  students  learning  to  their  potential? 

2.  Is  the  educational  system  supporting  student  learning 
efficiently  and  effectively? 

The  EQI  initiative  will  focus  on  developing  indicator  systems,  establishing 
procedures,  and  reporting  and  disseminating  the  information  to  educational 
constituencies  in  Alberta.  Figure  1 presents  the  expected  outcomes  for  these 
three  components. 
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Figure  1:  Expected  Outcomes  of  the  EQI  Initiative 
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The  fundamental  principle  of  the  EQI  initiative  is  that  no  single  indicator,  or  even 
group  of  indicators,  can  fully  describe  the  complexity  of  education.  The  proposed 
system  will  include  many  indicators,  measured  by  both  quantitative  and  qualitative 
methods,  for  selected  dimensions.  EQI  intends  to  include  indicators  organized  in 
logical  clusters,  measured  in  different  ways,  using  information  from  multiple  sources 
to  describe  education  in  such  a fashion  that  meaningful  interpretation  and  judgements 
can  be  made.  The  indicator  system  intends  to  enhance  information  about  education  for 
improved  action  in  planning,  policy  and  decision  making. 

A four-dimensional  model  of  education  was  developed  to  guide  the  direction  of  this 
initiative.  It  consists  of  partners  (schooling,  family  and  society),  conditions  (context, 
inputs  and  processes),  student  outcomes  (cognitive,  affective  and  behavioral)  and  time 
(grades  3,  6,  9 and  12)  (McEwen  and  Zatko,  1989). 

The  EQI  initiative  is  sponsoring  ten  concurrent  action  research  projects.  One  or  more 
school  jurisdiction(s)  within  the  zone  of  each  Regional  Office  of  Education  was 
identified  and  invited  to  participate.  Each  school  jurisdiction  prepared  a proposal 
based  on  the  Terms  of  Reference  and  submitted  it  for  approval.  The  Planning  and 
Policy  Secretariat  is  providing  funds  to  assist  these  jurisdictions  to  improve  their 
assessment  procedures  and  to  share  their  results  with  others  in  the  province.  The 
information  generated  from  the  ten  projects  will  assist  Alberta  Education  to  develop 
provincial  indicators.  The  interpretation  and  recommended  directions  of  the  local 
indicator  projects,  together  with  other  provincial  initiatives,  will  provide  a solid 
foundation  for  the  implementation  of  an  efficient  and  effective  information  system 
which  measures  the  success  of  the  educational  enterprise  in  the  province. 

Each  participating  school  jurisdiction  is  developing  a local  indicator  system  which 
includes  the  components  identified  in  Figure  1:  a set  of  indicators  (including  an 
interpretative  framework,  student  outcomes  and  points  of  reference),  methods 
(to  collect,  analyze  and  interpret  the  information),  and  a reporting  and  dissemination 
strategy  (to  inform  diverse  audiences  of  the  results).  Every  project  has  three  phases, 
each  of  approximately  one  year’s  duration.  The  first  year,  1989/90,  is  developmental 
and  the  subsequent  two  years  will  result  in  field  testing  the  prototype  sets  of  indicators 
and  methodological  procedures  and  then  refining  them. 

The  five  papers  present  different  aspects  of  the  principles  behind  EQI.  The  first  three 
papers  describe  the  involvement  of  the  community  in  determining  the  goals  and 
priorities  of  education  in  their  respective  school  jurisdictions.  The  last  two  papers 
describe  the  development  of  a broader  range  of  student  outcomes  to  include  the 
affective  and  social  domains.  The  five  papers  provide  a good  representation  of  the 
types  of  activities  taking  place  during  this  developmental  year  of  the  EQI  projects. 


Nelly  McEwen , Coordinator 
Educational  Quality  Indicators 
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The  Quality  Indicators  Strategy:  Involving  the  Community 


Grande  Prairie  Public  School  District  No.  2357  operates  ten  schools  in  the 
City  of  Grande  Prairie  and  has  a current  enrolment  of  4500  students.  Five  of  the 
present  schools  are  elementary  schools,  one  is  an  elementary-junior  high  school, 
one  is  a junior  high  school,  one  is  a composite  senior  high  school  and  one  is  an 
elementary-junior  high  school  which  offers  programs  and  services  to  150  multi- 
handicapped children  and  350  children  in  regular  school  classes.  A court  school  is 
also  operated  by  the  District  under  a contract  with  the  attorney-general. 

Extensive  Special  Education  services  are  offered  by  the  district,  including 
programs  for  trainable  mentally  handicapped,  educable  mentally  handicapped,  and 
all  levels  of  learning  disabilities.  Programs  for  gifted  and  talented  students  are  also 
offered  at  elementary,  junior  and  senior  high  schools. 

In  July  of  1989  the  District  entered  into  a contract  with  the  Minister  of 
Education  of  Alberta.  The  District  agreed  to  conduct  a research  project  for  the 
Minister  to  identify  indicators  that  could  be  used  to  determine  the  effectiveness  of  a 
school  system  in  developing  positive  student  outcomes  in  the  cognitive,  affective 
and  behavioural  domains. 

The  Grande  Prairie  Quality  Indicators  Strategy  has  as  its  main  emphasis  and 
focus  the  stakeholders  of  education  in  the  city  of  Grande  Prairie.  These 
stakeholders  are  defined  as  any  person  or  group  who:  receives  the  educational 
product  through  instruction;  uses  the  facilities  or  resources  of  the  district;  works  for 
the  district;  contributes  through  taxes  to  the  operation  of  the  district;  has  siblings 
attending  school  in  the  district;  or  hires  or  further  educates  the  students  of  the 
district.  The  project  team  has  identified  five  major  stakeholder  groups  including 
educators,  parents,  administrators,  students  and  the  public  and  has  developed  a 
strategy  to  gather  their  opinions  on  the  quality  of  education  in  the  city  of  Grande 
Prairie.  The  targeted  stakeholders  identified  quality  indicators  for  measuring  the 
effectiveness  of  the  school  district  and  its  schools.  This  paper  will  demonstrate  that 
while  some  of  the  initial  results  were  very  predictable,  some  of  the  indicators  were 
unique  to  the  Grande  Prairie  Public  School  District. 

Introduction 

The  Grande  Prairie  Public  School  District  became  involved  with  the 
Educational  Quality  Indicators  study  in  order  to  find  a way  to  measure  the  quality  of 
education  and  the  effectiveness  of  the  product  offered  to  students  in  the  District. 
The  related  literature  on  effective  schools,  high  performance  schools,  quality 
education,  and  accountability,  provided  the  research  team  with  many  questions 
regarding  reporting  of  school  effectiveness.  The  project  team  used  three  simple 
questions  to  guide  its  research  in  the  area: 

Who  wants  to  know? 

What  do  they  want  to  know? 

How  do  we  report  what  they  want  to  know? 

The  problem  was  not  only  to  identify  quality  indicators  but  also  to  develop  a 
method  to  measure  the  attainment  of  the  indicators  and,  finally,  to  report  the  results 
so  that  all  stakeholders  could  understand  and  use  the  information.  Particular 
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attention  was  given  to  the  Grande  Prairie  scene  to  determine  what  is  feasible  for 
Alberta  practice.  The  "feeling”  of  the  city  was  taken  from  stakeholders  who  have  a 
primary  interest  for  determining  the  effectiveness  of  schools.  An  exploratory  type 
of  research  was  employed.  In  discussing  the  functions  of  exploratory  studies, 
Selltiz,  Jahoda,  Deutsch,  and  Cook  (1959)  reported: 

Many  exploratory  studies  have  the  purpose  of  formulating  a problem 
for  more  precise  investigation  or  of  developing  hypotheses.  An 
exploratory  study  may,  however,  have  other  functions:  increasing 
the  investigator’s  familiarity  with  the  phenomenon  he  wishes  to 
investigate  in  a subsequent,  more  highly  structured,  study;  clarifying 
concepts;  establishing  priorities  for  further  research;  gathering 
information  about  practical  possibilities  for  carrying  out  research  in 
real-life  settings;  providing  a census  of  problems  regarded  as  urgent 
by  people  working  in  a given  field  of  social  relations,  (p.51) 

The  above  authors  suggest  using  the  following  methods  in  this  type  of  research:  a 
review  of  the  related  social  science  and  other  pertinent  literature  and  a survey  of 
people  who  have  had  practical  experience  with  the  problem  to  be  studied 

The  use  of  an  ex  post  facto  research  model  was  determined  by  the  following 
factors;  the  format  would  not  allow  any  direct  control  of  independent  variables;  the 
target  groups  could  not  be  manipulated  into  categories  which  would  be  required  for 
a true  experimental  or  quasi  experimental  research  design;  and,  the  targeted 
population  for  the  Grande  Prairie  study  already  possessed  opinions  and  experiences 
with  the  educational  system  before  the  research  started 

Further  support  to  conducting  an  ex  post  facto  research  project  was  offered 
by  Donald  Ary  (1972): 

We  contend  that  any  dissatisfaction  encountered  among  clients 
"deprived"  of  a new  program  is  a drop  compared  with  the  flood  of 
dissatisfaction  from  taxpayers  who  discover  that  millions  have  been 
spent  on  programs  that  lacked  a well  planned  method  for 
determining  whether  the  programs  actually  accomplished  anything 
or  not.  (p.317) 

The  research  team  believe  educators  have  relied  too  heavily  on  the  results  of 
standardized  tests  as  the  main  source  of  feedback  on  the  quality  and  effectiveness  of 
the  educational  process.  In  a paper  presented  to  the  UCLA  Center  for  the  Study  of 
Evaluation,  Eva  Baker  (1988)  states: 

Outcomes  like  student  achievement  test  scores,  college  admission 
rates,  or  dropout  figures  represent  the  easy  pan  of  indicators. 

Quality  indicators  should  also  take  into  account  input  variables  and 
measures  of  process,  (p.37) 

Baker  (1988)  goes  on  to  say: 

Achievement  testing  will  not  go  away,  and  for  good  reason. 
Students  and,  by  implication,  the  schools  to  which  they  go  must  be 
held  accountable  for  teaching  students  and  attempting  to  measure 
what  they  have  learned,  (p.28) 
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The  team  was  armed  with  many  opinions  from  educational  researchers  such  as 
Edmonds  (1978),  Brookover,  and  Lezotte  (1977)  on  measuring  school 
effectiveness.  The  related  literature  on  effective  schools,  high  performance  schools, 
quality  education,  and  accountability,  provide  researchers  with  many  questions  on 
the  reporting  of  school  effectiveness  (Frederick,  1987).  The  development  of  Lie 
strategy  for  the  Grande  Prairie  Public  School  District  study  involved  looking  at 
various  projects  conducted  in  the  United  States  (Codianni.  & Wilbur,  1983).  This 
research  provided  a solid  basis  for  understanding  the  work  that  had  been  done  in 
the  area  of  school  effectiveness  and  quality  indicators  of  that  effectiveness  at  the 
time  of  the  study.  The  team  had  to  establish  an  appropriate  method  for  measuring 
the  quality  indicators  of  the  Grande  Prairie  Public  School  District  as  defined  by  the 
targeted  stakeholders.  There  were  many  established  instruments  and  methods  which 
had  already  been  employed  by  educational  researchers  in  their  attempts  to  define 
criteria  for  measuring  educational  quality.  The  ’Summary  of  Quality'  Indicators 
study  conducted  by  the  Austin  Independent  School  District  cites  indicators  such  as 
student  achievement,  college  bound  students,  student  diversity,  basic  skills, 
attendance  and  drop  out  rates  in  comparing  the  effectiveness  of  their  district  with 
other  districts  (District,  1987).  The  review  of  the  research  on  quality  indicators  gave 
the  project  team  a comfortable  understanding  of  what  to  look  for  and  provided  a 
strategy  to  obtain  the  necessary  data. 

Phase  One:  District  Report  Card 

Procedures 

As  indicated  earlier,  the  first  step  in  phase  one  was  to  identify  the 
stakeholders  of  education  in  the  Grande  Prairie  Public  School  District  The  term 
stakeholder  also  had  to  be  identified  in  terms  that  would  enable  the  team  to  target 
appropriate  groups  for  input  Die  second  step  was  to  establish  a method  or  strategy 
to  identify  what  the  stakeholders  perceived  the  quality  indicators  of  the  school 
district  to  be.  In  order  to  attain  quality  information,  the  stakeholders  needed  to  be 
informed  of  the  overall  objectives  of  the  Grande  Prairie  Project.  The  ultimate 
purpose  of  the  study  w'as  determined  and  the  methodology  was  also  established  by 
the  project  team.  This  information  was  presented  to  each  stakeholder  group  as  an 
introduction  to  the  sessions  which  were  designed  to  obtain  their  opinions  on  quality 
indicators  of  education  for  the  district.  After  the  introduction  of  the  study,  the 
stakeholders  w'ere  asked  to  break  into  groups  to  brainstorm  their  quality  indicators 
of  an  effective  school  district.  After  listing  the  items  via  brainstorming,  the  groups 
were  asked  to  rate  their  indicators  from  most  important  to  least  important.  Once  the 
groups  listed  their  items  in  order  of  priority,  the  groups  shared  their  results  with 
one  another.  The  third  step  of  phase  one  was  to  analyze  the  data  obtained  from  the 
stakeholder  sessions.  The  data  was  placed  into  a data  base  which  identified  the 
stakeholder  group,  the  subgroup,  the  number  of  people  in  the  subgroup,  the  date, 
and  the  prioritized  quality  indicators  of  that  subgroup.  All  identified  quality 
indicators  w-ere  then  placed  on  a spreadsheet  and  assigned  a value  of  five  points  for 
a number  one  rating,  four  points  for  second  place,  three  points  for  third  place,  two 
points  for  fourth  place  and  one  point  for  fifth  place.  The  spreadsheet  data  is 
categorized  into  individual  stakeholder  groups  and  a blended  group  of  all 
stakeholders.  (See:  Table  1) 
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Table  1 

Number  of  Stakeholders  Involved  in  the  Grande  Prairie 
Educational  Quality  Indicators  Strategy 


Groups 

Number 

Parents 

35 

Public 

36 

Students 

241 

Instructional  Staff 

229 

Non-Instructional  Staff 

84 

Administrators 

36 

Total 

661 

The  project  team  will  report  back  to  all  stakeholders  surveyed  during  phase  one  to 
solidify  the  criteria  identified  during  the  stakeholder  sessions.  After  the 
identification  of  stakeholders,  the  project  team  recruited  volunteers  to  represent 
various  community  groups  on  a Quality  Indicators  Steering  Committee.  The  team 
meets  with  this  committee  every  three  months  to  update  them  on  the  progress  and  to 
obtain  feedback  and  direction  on  the  project  course. 

Preliminary  Results 

The  team  is  currently  analyzing  the  data  and  deciding  how  best  to  report  the 
findings.  There  have  been  some  interesting  results  from  the  first  phase  of  the 
surveys.  All  groups  thus  far  have  identified  student  achievement  as  one  of  the 
indicators  of  an  effective  school  district.  There  are  varying  opinions,  however,  on 
the  meaning  of  student  achievement  The  sessions  conducted  with  the  public  sector 
indicates  that  items  such  as  employability  of  graduates,  respect,  work  ethic, 
attitudes  and  self  esteem  are  considered  to  be  measures  of  student  achievement. 
Parent  groups,  educators  and  students  hold  standardized  test  scores,  and  district 
comparisons  on  provincial  tests  as  measures  of  student  achievement.  Research  in 
the  United  States  indicates  administrative  leadership  as  a strong  force  in  the  measure 
of  effectiveness.  The  initial  results  of  the  Grande  Prairie  survey  indicate  that  few 
stakeholder  groups  rate  Leadership  in  the  top  five  indicators.  Some  groups,  such  as 
educators,  do  not  mention  leadership  at  all.  The  end  goal  of  phase  one  of  the  study 
is  to  develop  a report  card  for  the  district  which  will  reflect  the  quality  indicators 
identified  by  the  stakeholders  in  phase  one.  To  date,  the  school  district  has 
published  a report  card  on  its  performance  based  on  the  measurable  criteria  they 
currently  possess. 
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Reporting  performance  in  this  context  was  very  well  received  by  the  school 
board  and  their  enthusiasm  provided  much  encouragement  to  the  project  team  that 
they  have  chosen  realistic  and  valuable  objectives  for  the  project.  The  purposes  for 
reporting  on  the  quality  indicators  of  an  effective  school  district  are  two-fold. 
Firstiy,  it  provides  all  stakeholders  in  education  in  Grande  Prairie  with  meaningful 
feedback  on  the  educational  process  and  its  results.  This  feedback  will  report  on 
criteria  that  is  important  and  has  been  identified  by  the  people  who  have  a stake  in 
what  happens  in  the  district.  It  will  serve  as  a common  basis  for  discussions  and 
questions  on  the  districts’  performance.  The  process  of  developing  and  reporting  on 
these  quality  indicators  will  result  in  more  knowledgeable  stakeholders.  More 
knowledgeable  stakeholders  equate  to  more  knowledgeable  questions  and  solutions 
to  concerns.  The  feedback  will  also  help  foster  a community  or  team  approach  to 
education  in  the  Grande  Prairie  Public  School  districts.  Stakeholders  can  truly 
become  partners  in  the  business  of  educating  youth. 

Secondly,  the  district  report  card  will  serve  as  a blueprint  for  strategic 
planning  to  improve  the  delivery  of  education  in  Grande  Prairie.  All  participants  in 
the  study  understand  there  is  nothing  that  can  be  done  about  past  performance.  The 
district  report  card  will,  however,  provide  us  with  a basis  to  target  areas  for 
improvement  and  a means  of  measuring  whether  or  not  the  planned  strategy  is 
working. 

It  is  very  clear  that  the  process  of  developing  the  report  card  will  be  just  as 
valuable  as  the  planned  outcome.  The  educational  industry  can  no  longer  make 
decisions  and  function  in  isolation  from  their  stakeholders.  Vicki  Bowers  (1990) 
makes  a strong  argument  for  more  communication  and  feedback  to  stakeholders, 

The  tragic  thing  is  that  schools  teach  exciting,  useful  and  important 
things  to  youngsters.  By  not  openly  imparting  that  information  to 
parents,  schools  deny  the  people  who  would  be  their  best  supporters 
the  chance  to  speak  knowledgeably.  And  that's  self  defeating, 
because  well  informed  parents  can  counter  the  radical  fringe  more 
effectively  than  can  school  public  relations  people,  (p.40) 

Phase  Two:  School  Report  Cards 

The  second  phase  of  the  study  will  involve  the  development  of  school  report 
cards.  Schools  will  be  asked  to  volunteer  to  work  on  the  development  of  individual 
report  card  which  will  reflect  the  opinions  of  their  school's  stakeholders.  The 
process  of  gathering  these  opinions  will  emulate  the  methods  used  in  developing  a 
district  report  card.  The  project  team  will  work  with  the  individual  schools  to 
identify  and  plan  a time  line  to  reach  their  stakeholders.  It  is  expected  that  the 
quality  indicators  identified  by  the  individual  schools'  stakeholders  will  vary. 

In  presenting  phase  one  of  the  study  to  various  stakeholder  groups, 
differences  were  witnessed  in  quality  indicators  as  the  study  was  presented  to 
various  school  communities.  An  example  of  this  difference  was  in  the  results  from 
educators  in  two  separate  elementary  schools.  Elementary  school  A listed  indicators 
such  as  program  equity,  special  programs,  student  motivation,  and  teachers 
involvement  in  decisions  while  Elementary  school  B listed  different  items  such  as; 
mainstreaming,  resources,  inservice,  and  Board/Teacher  relations. 
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There  will  be  an  existing  data  bank  from  phase  one  of  the  study  that  can  be 
used  as  a basis  for  discussion  among  the  school  stakeholder  groups.  For  example, 
in  the  survey  of  the  Parent  stakeholders,  the  presentations  were  to  existing  school 
Parent  Advisory  committees.  Each  committee  represented  a specific  school  in  the 
district.  Similarly,  the  instructional  and  non-instructional  staff  were  contacted  on  a 
school  by  school  basis.  The  project  team  will  contact  as  many  stakeholders  as 
possible  using  the  nominal  group  technique  described  in  phase  one.  Further  data 
gathering  may  have  to  be  accomplished  using  questionnaires  or  some  sort  of  mass 
media  campaign. 

The  resulting  school  report  card  from  phase  two  of  the  study  will  include 
common  items  identified  by  all  schools  as  well  as  the  indicators  specific  to 
individual  schools.  The  major  work  for  the  project  team,  once  the  quality  indicators 
have  been  identified,  will  be  the  development  of  tools  to  measure  the  quality 
indicators  and  to  effectively  communicate  the  results  to  the  stakeholders.  As  the  data 
gathering  process  is  still  in  its  early  stages,  the  number  of  indicators  included  in 
each  report  card  cannot  be  determined.  At  the  conclusion  of  phase  two,  each  school 
will  be  provided  with  the  means  to  gather  quality  indicator  information  on  an  annual 
basis. 


It  is  expected  that  these  annual  report  cards  will  serve  stakeholders  in 
determining  the  existing  strengths  and  weaknesses  of  their  individual  schools  and 
also  serve  as  an  instrument  to  collectively  plan  and  improve  the  quality  of  their 
schools.  Initial  presentations  during  phase  one  of  the  study  have  already  produced 
schools  who  wish  to  volunteer  to  develop  report  cards  during  phase  two  of  the 
study.  Phase  two  of  the  study  will  also  see  the  project  team  identify  a comparable 
school  district  to  pilot  the  gathering  and  reporting  of  quality  indicators.  This  school 
district  can  provide  the  study  with  valuable  feedback  on  the  usefulness  of  this 
process  in  schools  and  districts  other  than  Grande  Prairie.  It  will  also  serve  to 
identify  common  indicators  and  disparities  which  exist  from  one  school  district  to 
another. 


Phase  Three:  Annual  Report  Cards 

The  final  year  of  the  Grande  Prairie  study  will  be  the  publication  of  annual 
repon  cards  on  the  school  district  and  each  school  within  the  district.  The  format  of 
these  report  cards  will  be  the  same  but,  the  identified  quality  indicators  of  each  will 
reflect  the  differing  opinions  of  each  schools’  stakeholders.  The  methodology  used 
to  gather  and  report  on  these  indicators  will  be  formalized  so  the  data  can  be 
gathered  and  reported  on  a yearly  basis.  The  final  product  will  also  serve  as  a basis 
for  setting  goals,  planning  change  and  measuring  progress  of  each  school  from  year 
to  year.  Administrators  and  teachers  need  not  waste  valuable  time  gathering  data 
which  has  no  significant  value  to  the  district  or  any  other  stakeholders.  Education  is 
unique  in  industry  for  its  lack  of  feedback  to  its  stakeholders.  David  Kearns  says, 
"I  can't  think  of  any  other  single  sector  of  American  society  that  has  absorbed  more 
money  by  serving  fewer  people  with  steadily  declining  service”  as  cited  by  Mann 
(1990,  p.  26). 

Kearns  goes  on  to  say,  "Teaching  is  the  only  profession  I know  of  that  if 
you  do  well,  nothing  good  happens  to  you,  and  if  you  do  poorly,  nothing  bad 
happens  to  you”  (Mann,  1990,  p.  26).  The  statements  of  David  Kearns 
demonstrates  the  frustration  exhibited  by  the  public  in  dealing  with  education.  A 
need  has  been  demonstrated  for  more  innovative  methods  of  providing  stakeholders 
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with  feedback.  Stakeholders  can  no  longer  rely  on  the  convenience  of  standardized 
test  scores  and  comparisons  with  national  averages  as  the  measures  of  the  quality  of 
the  educational  systems.  In  an  article  in  the  Executive  Educator,  John  G.  Weiss  and 
Arnold  F.  Fege  say,  "Ultimately,  instead  of  measuring  the  success  of  the  public 
education  system,  standardized  tests  will  dictate  what  is  taught" . Weiss  and  Fege 
indicate  in  their  article  that  what  is  measured  becomes  a key  factor  in  determining 
behavior.  "So  there  is  every  reason  to  believe  the  introduction  of  state  by  state 
comparisons  of  standardized  test  scores  will  influence  the  public's  perception  of 
each  state's  educational  system."  These  changes,  the  authors  believe,  will  result  in 
the  public  demanding  changes  that  would  eventually  harm  schools.  The  authors 
state  that  the  test  were  never  meant  to  be  a vehicle  of  change,  yet  that  is  what  they 
are  capable  of  becoming.  In  closing  the  authors  say,  "Unless  we  establish 
safeguards,  we  might  find  that  standardized  tests  have  become  the  tail  that  wags  the 
dog"  (Weiss,  & Fege,  1988,  p.14). 

Educators,  must  begin  to  listen  to  communities  and  fellow  educators  and 
repon  on  information  that  will  provide  a basis  for  change  in  education  that  will  be 
positive  and  in  the  best  interests  of  all  stakeholders  in  education.  Change  is  not  an 
end  product  but  an  ongoing  process.  The  ongoing  process  will  reflect  the  changes 
in  society  and  the  job  market  to  continuously  educate  youth  with  information  that 
will  be  valuable  to  them  for  life.  Education  cannot  be  content  with  the  philosophy 
that  prevailed  in  the  past  where  educators  were  providing  students  with  knowledge 
and  skills  which  are  not  required  by  industries  of  today. 
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A 


A Collaborative  Model  for  School  and  Program  Evaluation 

In  recent  years  there  has  been  a marked  increase  in  the  emphasis  on  carrying  out 
program  and  school  evaluations.  In  Lethbridge  School  District  #5 1 , and  indeed 
throughout  much  of  Alberta,  the  approach  used  has  been  predominantly  "top  down" 
with  the  planning  and  procedures  carried  out  almost  totally  by  personnel  from  the 
district  central  office  and  assisted  by  external  resource  persons.  A meta-evaluation 
concluded  that  limited  outcomes  were  being  realized  by  the  evaluations  since  the 
teachers  and  school-based  administrators  were  not  actively  involved  (except  for 
having  the  process  "done  to"  them)  and  hence  gained  little  from  the  exercise.  A 
study  of  die  related  literature  and  of  procedures  followed  in  other  locations  led  to  the 
proposal  for  a "collaborative  model"  and  the  empowering  of  school-based  personnel 
to  be  actively  and  professionally  involved  in  their  program  and  school  evaluations. 
The  use  of  "educational  quality  indicators"  serves  as  a key  component  of  the 
proposed  model. 


Introduction 

"Serious  conversations  about  education  matters  must  precede  the 
evaluation  of  school  organizations , personnel , students  and 
curriculum . It  is  through  these  conversations  that  we  will  be  able 
to  get  our  ideas  straight  about  quality  in  education  and  about  how 
to  reach  it. " (Common,  1987,  p.  1 1) 

In  recent  years,  evaluation  of  teaching  and  of  teachers  has  received  in-depth 
study  but  little  effort  seems  to  have  been  put  into  whether  the  models  and 
procedures  used  for  the  relatively  new  focus  on  school  and  program  evaluations  are 
consistent  with  the  findings  of  educational  research.  Because  the  stated  goal  of 
evaluation  is  almost  always  school  improvement,  there  is  a need  to  consider  and 
utilize  the  research  on  quality  indicators,  school  effectiveness,  and  school 
improvement 

This  study  was  based  on  the  perceived  need  for  an  analysis  of  the  procedures 
being  used  in  one  Alberta  school  jurisdiction  to  carry  out  program  and  school 
evaluations,  and  for  the  development  of  a new  model  based  on  the  current  research 
and  literature  in  the  areas  of  indicators  of  effectiveness,  quality,  and  school 
improvement  The  need  to  evaluate  schools  and  programs  was  succinctly  described 
by  Goodlad  (1984,  p.l):  "to  survive,  an  institution  requires  from  its  clients 
substantial  faith  in  its  usefulness  and  a measure  of  satisfaction  with  its 
performance." 

Background 

In  the  province  of  Alberta,  as  in  other  Canadian  provinces,  and  a number  of 
American  states  there  has  been  an  increasing  emphasis  and  expectation  from 
governments  that  school  jurisdictions  will  develop  and  carry  out  formalized 
procedures  for  evaluation,  including  the  evaluation  of  students,  teachers,  programs, 
schools,  and  school  systems. 

This  descriptive  research  project  involved  carrying  out  a case  study  of  the  model 
of  school  and  program  evaluation  employed  in  Lethbridge  School  District  #5 1 (LSD 
#51),  a medium-sized  Alberta  jurisdiction  of  approximately  8,000  students  and  450 
professional  staff.  The  Lethbridge  model  involved  the  use  of  a large  team  (up  to 
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fifty  five  people)  from  the  school  district  central  office,  other  schools,  Alberta 
Education  offices,  and  the  University  of  Lethbridge  spending  up  to  three  weeks 
time  on  site  for  the  school  evaluation  process.  A common  concern  was  whether  the 
model  (which  was  extremely  expensive  in  terms  of  professional  time  and  expertise) 
was  producing  payoffs  in  professional  growth  and  development  Perhaps  the  most 
significant  concern  was  with  the  problem  of  implementing  the  recommendations 
emanating  from  the  evaluation  report;  members  of  the  district  support  staff 
(consultants,  coordinators,  and  superintendents)  were  so  heavily  involved  with 
preparing  for  and  carrying  out  the  evaluations  that  they  had  little  or  no  time  to 
provide  follow-up  or  assistance  with  implementing  the  recommendations  made  in 
the  previous  ones. 


Purpose  and  Significance  of  the  Study 

'The  most  important  purpose  of  evaluation  is  not 
to  prove  but  to  improve.0  Stufflebeam  and  Shinkfield  (1985,  p.  151). 

Administrators  and  boards  need  assurance  that  their  models  of  school  and  program 
evaluation  are  consistent  with  educational  research  and,  therefore,  likely  to  bring 
about  increased  effectiveness  and  school  improvement  as  a result  of  the  investments 
of  professional  time,  expertise,  and  budget  dollars.  Stufflebeam  and  Shinkfield 
(1985)  affirmed  that  if  evaluations  are  to  be  useful  and  provide  proper  direction  and 
guidance,  'the  evaluations  themselves  must  be  sound'  (p.  1 83). 

In  carrying  out  this  study,  the  following  research  questions  were  answered: 

1.  Is  the  current  LSD  #51  model  for  program  and  school  evaluation  consistent  with 
the  literature  on  indicators  of  effectiveness,  quality,  and  school  improvement? 

2.  Are  the  procedures,  instruments,  and  data  used  in  the  LSD  #5 1 model  valid  and 
reliable? 

3.  Does  the  current  literature  suggest  characteristics  or  criteria  of  more  effective  and 
efficient  models  for  program  and  school  evaluation,  in  terms  of  indicators  of 
effectiveness,  quality,  and  improvement? 


Review  of  Related  Literature 

'Put  on  one  pair  of  glasses  and  our  schools  appear  to  be  the  worst 
of  places.  Put  on  another  and  they  appear  to  be  the  best0  (Goodiad, 
1984,  p.  10) 

The  following  major  areas  of  educational  literature  were  reviewed  as  a basis  for 
this  study  and  the  proposed  model 

(1)  school  and  program  evaluation  - theory  and  practice, 

(2)  effective  schools  research,  (3)  indicators  of  effectiveness  or  quality,  (4) 
school  improvement  literature,  and  (5)  evaluation  models,  criteria,  and 
standards. 
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Purposes  of  school  and  program  evaluations  were  succinctly  stated  by 
Stufflebeam  (1971,  p.  1 57)  as  "'not  to  prove  but  to  improve!'  Other  items  from  the 
literature  revealed  consistency  in  the  view  that  schools  and  programs  should  be 
evaluated.  Both  Eisner  (1985b)  and  Goodlad  (1984)  stressed  the  lack  of  public 
confidence  in  our  schools.  Eisner  (1985b,  p.  1)  suggested  that,  'It  is  possible  that 
our  entire  educational  system  is  near  collapse."  Common  (1987)  noted  an  apparent 
contradiction  between  what  Maclean's  magazine  referred  to  as  a crisis  in  Canadian 
and  American  schools  (Finlayson,  1985),  and  the  many  Canadian  research  studies 
and  polls  showing  that  teachers,  parents,  and  students  regard  our  schools  as  doing  a 
reasonably  good  job.  Common  (1987)  concluded:  "There  is  rightfully  a justification 
for  careful  evaluation  of  the  curriculum,  the  professionals  and  students,  and  the 
organization  that  constitutes  public  education"  (p.9). 


Meta-Evaluation : Evaluating  the  Evaluation 

"Quis  custodiet  ipsos  custodes  ?* 

The  above  reference  by  Morgan  (1986,  p.  26)  to  'who  guards  the  guards?' 
serves  as  a reminder  that  evaluations  themselves  need  to  be  evaluated,  and  that  those 
doing  the  evaluations  need  to  be  competent  and  trustworthy.  He  asserted  that  no 
one  should  be  immune  from  evaluation,  especially  not  the  evaluators  themselves. 
Levin  (1983,  p.  1 1)  concurred  and  noted  "Given  the  amount  of  time,  effort  and 
money  which  may  be  involved  in  an  evaluation,  and  the  importance  which  its  results 
may  have,  school  districts  need  to  be  sure  that  evaluations  do  ’deliver  the  goods.’" 

Common  (1987,  pp.  11-15)  strongly  put  forth  a number  of  concerns  about 
evaluation  as  it  is  now  done.  "When  evaluation  is  a public  action,  it  becomes  a 
political  activity  in  which  power  and  powerless  become  central.'  She  stated  her 
opposition  to  external  models  for  evaluation  because  they  are  costly,  contentious, 
lower  teacher  morale,  and  generate  data  which  may  be  used  very  little.  She  stated 
further  that  they  promote  an  "impoverished  view  of  education  and  deskill  the  teacher 
at  a time  when  his  (or  her)  professional  skills  are  more  important  than  ever."  Her 
view  is  that  external  models  of  evaluation  may  stop  curriculum  innovation  and,  at 
best,  are  unlikely  to  make  it  start 

In  considering  the  nature  of  evaluations,  and  what  should  take  place, 
Stufflebeam  and  Shinkfield  (1985,  pp.  70-74)  reported  that  Ralph  W.  Tyler 
(generally  recognized  as  the  founder  of  educational  evaluation)  considered  that 
evaluation  should  determine  the  congruence  between  performance  and  objectives. 
This  approach  laid  the  foundation  for  an  objective-oriented  style  of  evaluation  as  far 
back  as  1942. 

In  considering  the  "how"  of  evaluation,  Eisner  (1979,  p.  267)  expressed  his 
strong  view  that  procedures  and  criteria  used  to  evaluate  students,  teachers,  and 
school  administrators  "have  profound  effects  on  the  content  and  form  of  schooling." 
He  seemed  to  lend  considerable  support  to  the  argument  for  school-based  evaluation 
with  his  statement  that,  "The  school  is  the  basic  unit  of  educational  excellence"  (p. 
280).  Goodlad  (1984,  p.  3 1)  offered  some  related  strategic  advice  when  he  stated 
that  "efforts  at  improvement  must  encompass  the  school  as  a system  of  interacting 
parts,  each  affecting  the  others." 

Scriven  and  Stufflebeam  discussed  "meta-evaluation"  and  the  need  for  assessing 
evaluation  criteria,  processes,  and  techniques  in  Standards  for  Evaluations  of 
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Educational  Programs , Projects , and  Materials  (1981).  Scriven  (1976)  stressed  the 
fact  that  evaluators  have  a professional  obligation  to  ensure  that  their  evaluations  are 
subjected  to  competent  evaluation.  He  emphasized  that  it  could  be  formative  or 
summative  and  could  include  use  of  the  "Key  Evaluation  Checklist"  This  checklist 
reflects  Scriven’s  views  that  evaluation  involves  multiple  dimensions  and  should 
employ  multiple  perspectives,  utilizes  multiple  levels  of  measurement  and  makes 
use  of  multiple  methods. 


Effective  Schools  Research 

'It  is  vastly  easier  to  describe  effectiveness 

than  to  achieve  it.'  Leithwood  and  Fullan  (1984,  p.  12) 

Research  on  effective  schools  had  its  genesis  somewhat  in  response  to  the  well- 
publicized  works  of  Coleman  (1966)  and  Jencks  (1973).  They  held  very 
pessimistic  views  that  schools  could  do  very  little  to  reduce  apparent  inequality 
among  children  in  terms  of  achievement 

Edmonds’  work  (1979)  showed  that  some  schools  succeed  where  others  fail 
and  identified  five  characteristics  of  successful  schools:  principals  who  provide 
strong  administrative  and  instructional  leadership,  high  expectations  that  all  students 
can  and  will  learn,  a school  climate  that  stimulates  learning  (beginning  with  a 
building  that  is  orderly  and  quiet  without  being  repressive),  students  and  staff  who 
believe  basic  skills  are  urgently  important,  and  a continuous  system  of  monitoring 
student  progress. 

The  major  importance  of  school  leadership  was  reiterated  by  Goodlad  (1979) 
who  asserted  that  the  principal  is  the  central  figure  in  the  attainment  of  a quality 
school  program.  Simultaneously,  in  Britain,  Rutter  et  al.  (1979)  identified  that  it 
was  the  "ethos"  or  tone  of  the  school  that  really  mattered  and  identified  the 
characteristics  which  yielded  desired  results  as:  firm  leadership  and  teacher 
involvement  in  decision-making,  positive  climate,  school  organization  and  teacher 
skills,  high  level  of  expectations  for  learning  and  behavior,  frequent  use  of  rewards 
and  praise,  emphasis  on  learning,  appearance  and  comfort  of  school  environment, 
and  opportunities  for  student  participation  and  responsibility. 

Over  the  decade  of  the  1980s,  a body  of  literature  known  as  "effective-schools 
research"  developed,  producing  a remarkably  consistent  set  of  findings.  Purkey 
and  Smith  (1983),  after  a wide  review  of  studies,  concluded  that  the  findings  of 
recent  school-effects  research  contradict  the  conclusions  of  Coleman  (1966),  Jencks 
(1973),  and  others.  Schools  can  make  a difference! 

Other  related  research  supported  the  above  conclusions.  Shulman  (1984,  p.  6) 
defined  effective  schools  as  "those  that  are  educative  settings  for  teachers."  A four 
-year  study  in  British  elementary  schools  by  Mortimore  and  Sammons  (1987) 
concluded:  "much  of  the  variation  between  schools  can  be  accounted  for  by 
differences  in  school  policies  and  practices  within  control  of  the  principal  and 
teachers"  (p.  4).  Although  not  writing  about  teaching,  Peters  and  Waterman  (1982) 
found  that  the  best-run  companies  were  outstanding  not  because  they  were  able  to 
recruit  and  hire  extraordinary  people  but  because  they  were  able  to  motivate  average 
employees  to  extraordinary  dedication  and  performance. 
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Of  significant  interest  is  the  observation  from  the  literature  that  not  all 
researchers  and  writers  on  the  topic  are  committed  to  the  effective  school 
philosophy;  a number  of  criticisms  have  emerged.  GUckman  (1987,  p.  624)  argued 
that  schools  and  researchers  have  failed  to  distinguish  between  good  and  effective 
schools  and  that  'The  ’effectiveness’  movement  is  unnecessarily  restricting  the 
curriculum,  narrowing  the  teaching  approach  to  direct  instruction,  and  controlling 
teachers." 

Cuban  (1984)  also  identified  problems  with  the  effective-schools  research 
including  that  no  one  knows  how  to  create  effective  schools,  the  language  is  fuzzy, 
effectiveness  is  constricted  to  test  results,  and  most  research  was  done  in 
elementary  schools.  Stedman  (1987,  p.  222)  concurred  and  argued,  'If  we 
continue  to  evaluate  schools  with  narrowly  designed  standardized  tests  then  we  will 
get  narrowly  structured  schools  to  impart  such  skills.' 

Stedman  (1988,  p.  442)  advocated  considering  an  alternative  approach  based  on 
cultural  pluralism,  academically  rich  programs,  personal  attention  to  students,  and 
shared  governance  with  parents  and  teachers.  Fullan  ( 1985,  p.  4 14)  cautioned  that 
'Nothing  would  be  worse  than  establishing  a grand  scheme  putting  all  schools  in 
the  district  through  the  paces  of  developing  effective  school  plans.' 

It  is  evident  that  evaluations  of  programs  and  schools  must  look  well  beyond 
the  criteria  commonly  associated  with  the  effective-schools  movement  Evaluations 
must  be  broadly -based  and  multi -dimensional 


Indicators  of  Effectiveness  or  Quality 

A new  body  of  literature  is  beginning  to  evolve  and  offers  promise  as  a means 
of  describing  effectiveness  and  quality  it  education.  The  development  of 
'indicators'  is  providing  a new  focus  and  emphasis  in  educational  studies  and  in 
schools. 

Common  (1987,  pp.  10-11)  described  quality  in  education  as  a 'mental 
construct""  and  elaborated  that 

"What  we  deem  to  be  quality  in  education  and  the  criteria  we  use  to  judge  it 
is  as  much  a product  of  the  morality  and  other  values  of  the  times  as  it  is  the 
product  of  what  we  know  about  what  we  are  doing.  The  secrets  of  what  we 
deem  quality  in  education  to  be  lie  not  in  the  measurements  we  take,  but  in  the 
judgements  of  worth  that  we  make  about  those  measurements.' 

Mumane  and  Pauly  (1988)  stressed  the  importance  of  developing  multiple 
indicators,  and  Kaagan  and  Smith  (1985)  pointed  out  that  indicators  provide 
information  about  the  health  of  a school  system  but  cautioned  that  a common  set  of 
indicators  would  almost  certainly  increase  the  move  toward  centralization.  A 
British  Columbia  study  by  Coleman  (1986)  introduced  an  element  called 
school/district  'ethos'  or  goodness  as  a potential  predictor  of  school  effectiveness. 

In  the  view  of  Porter  (1988,  pp.  504-505),  the  use  of  indicators  could  lead  to 
more  centralized  control  over  the  education  system  in  terms  of  what,  how,  by 
whom,  and  to  what  standards  things  are  taught  His  pessimism  was  offset  by  the 
statement  that  'designing  and  implementing  a system  of  educational  indicators  could 
become  one  of  the  first  steps  toward  the  meaningful  participation  of  teachers  in 
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setting  educational  policy"  (p.  506).  Porter  foresaw  the  possibility  of  enhancing  the 
professional  status  of  teachers  if  they  participated  in  the  design  of  the  indicator 
system  but  was  also  cognizant  of  the  danger  that  participation  in  the  development 
process  "could  result  in  ...  the  lowest  common  denominator  of  current  practice" 
(p.  508). 

The  Colorado  Department  of  Education  ( 1 982)  presented  indicators  of  quality  in 
12  categories,  with  a total  of  42  indicators:  curricular  congruence,  assessment, 
leadership  of  principal,  high  expectations,  school-wide  norms,  values,  practices 
and  policies,  school  climate  factors,  monitoring  and  feedback  of  student  progress, 
time  on  task,  organization  and  management  of  the  instructional  setting,  instructional 
effectiveness,  and  parent  and  community  involvement  This  list  bears  a striking 
similarity  to  the  effective-schools  criteria  discussed  earlier  in  the  review  of  literature 
and  should  be  of  considerable  use  in  meeting  the  purposes  of  this  study. 

In  both  Alberta  and  British  Columbia  considerable  efforts  are  underway  using 
the  concepts  of  indicators  of  quality.  The  British  Columbia  Ministry  of  Education 
(1986,  p.  16)  presented  an  evaluation  model  of  three  components:  goal  statements, 
quality  indicators,  and  an  interpretive  framework  (the  judgments  and  views  of 
professionals  and  members  of  the  public  who  are  knowledgeable  about  education). 
McEwen  and  Zatko  (1989,  p.  13),  on  behalf  of  Alberta  Education,  provided 
examples  of  indicators  of  students’  cognitive,  affective,  and  behavioral  outcomes. 
The  cognitive  indicators  which  they  suggested  as  examples  included  achievement 
on  provincial  and  standardized  tests,  program  participation,  diploma  type, 
graduation  rate,  dropout  rate,  and  scholarships.  Affective  indicators  included 
satisfaction  with  schooling,  self-esteem,  motivation,  values,  and  attitudes  toward 
school,  subjects  and  work.  Their  indicators  of  behavioral  outcomes  were  in  two 
categories,  physical  and  social.  The  indicators  of  physical  behaviors  were  fitness, 
health  (nutrition,  hygiene),  and  freedom  from  substance  abuse,  social  diseases, 
pregnancy,  and  suicide.  The  indicators  of  social  behavior  included  the  set  of 
desirable  personal  characteristics  adopted  by  the  government  of  Alberta  - 
ethical/moral,  intellectual,  and  social  and  personal. 

The  Ministere  de  1’ Education,  Gouvemement  du  Quebec  (1989),  published  a set 
of  indicators  as  one  means  of  responding  to  the  demand  for  accountability  in  public 
administration.  Their  indicators  were  presented  in  five  categories: 

(1)  Financial  resources,  including  spending  in  relation  to  GNP,  school 
board  spending  per  student,  student-teacher  ratio,  and  average  teacher  salaries. 

(2)  Progress  through  school,  measured  by  numbers  reaching  and 
completing  secondary  school,  falling  behind,  or  dropping  out  of  school. 

(3)  Evaluation  of  learning,  such  as  secondary  school  examination  results  by 
sex,  school  system,  language  of  instruction,  type  of  education,  ana  considering 
regional  disparities,  and  subjects. 

(4)  Secondary  school  graduates  in  terms  of  numbers  and  types  of  diplomas, 
numbers  going  on  to  college,  and  numbers  joining  the  work  force. 

(5)  Adult  education  including  spending  by  board,  and  numbers  of  adult 
graduates. 

In  the  United  States  of  America,  The  Office  of  Educational  Research  and 
Improvement  (1987)  presented  the  following  "outcomes'  as  indicators:  reading 
performance,  writing  performance,  college-entrance  examination  scores,  high 
-school  completion  by  race  and  ethnicity,  literacy  skills  of  young  adults,  and 
participation  of  high-school  graduates  in  postsecondary  education.  The  "resources" 
listed  as  indicators  were  expenditures  per  pupil,  pupil/teacher  ratios,  and  teacher 
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salaries.  The  "context’  indicators  were  school  enrollment  (by  age  groups),  aspects 
of  home  environment,  student  drug  and  alcohol  abuse,  teacher  job  satisfaction, 
school  problems  as  seen  by  teachers  and  the  public,  public  opinion  ratings  of 
schools,  and  state  high  school  graduation  requirements. 

Some  cautions  are  warranted  by  the  indications  of  serious  defects  as  pointed  out 
by  Winters  ( 1985)  in  the  California  high  school  quality  indicator  program.  She 
warned  that  the  quality  indicators  are  uninterpretable,  and  can  be  misleading.  It  is 
the  view  of  the  researcher  that  this  need  not  be  true  since  the  indicators  are 
developed  by  teachers  and  should  be  written  in  such  a way  that  they  can  be 
interpreted  easily. 


School  Improvement 


'Change  is  a process,  not  an  event!' 

(Hall  and  Loucks,  1977,  p.  17). 

Since  the  goal  of  school  and  program  evaluation  is  the  improvement  of  schools, 
it  is  important  to  consider  the  literature  on  improvement  Close  similarities  exist 
between  the  findings  here  and  to  the  research  on  effective  schools. 

Leithwood  and  Fullan  (1984)  proposed  six  strategies  for  increasing  the  chance 
of  successful  change:  continuous  professional  development,  increasing  principal 
effectiveness,  school  planning,  developing  policies  with  a view  to  their 
implementation,  using  standard  operating  procedures,  and  building  systematic 
problem-solving  procedures. 

Fullan  (1985)  went  even  further  and  presented  a set  of  school-level  strategies. 
This  included  developing  a plan,  investing  in  local  facilitators,  allocating  resources, 
selecting  schools,  and  deciding  on  the  scope  of  project,  developing  the  principal’s 
leadership  role,  focusing  on  instruction,  stressing  ongoing  staff  development, 
ensuring  information  gathering  and  use,  planning  for  continuation  and  spread,  and 
reviewing  the  capacity  for  future  change. 

Lezotte  and  Bancroft  (1985)  noted  that  successful  local  school  improvement 
programs  have  in  common  a focus  on  a single  school,  a building-based 
improvement  team,  a longer-term  orientation  (three  to  five  years)  in  planning  and 
implementation,  and  are  organized  around  the  concept  of  the  effective  school  as  in 
research. 

Wood,  Freeland,  and  Szabo  (1985)  noted  the  present  thrust  for  school 
improvement  differs  from  the  past  in  that  the  target  is  no  longer  the  district  or 
individual  staff  member,  but  the  school.  Their  conclusions  were  that  the  primary 
method  for  achieving  improvement  is  not  curriculum  development  but  staff 
development,  that  the  source  of  improvement  is  not  just  intuition  but  research  on 
effective  schools  and  effective  instructional  practices,  and  that  planning  is  no  longer 
year-to-  year  responding  only  to  immediate  needs,  concerns  and  problems,  but  is 
proactive,  long  range,  and  systematic. 

Naisbitt  (1982,  p.  3)  appeared  to  lend  support  to  this  school-based  model  of 
improvement  with  his  statement:  "Trends  are  bottom-up,  fads  top-down."  He 
advocated  moving  away  from  the  specialist  who  is  soon  obsolete  to  the  generalist 
who  can  adapt  to  a 'high- tech/high -touch"  world.  Although  he  was  speaking 
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mainly  about  businesses,  Naisbitt’s  ideas,  ''long-range  plans  must  replace  short- 
term profit,'  could  apply  equally  to  schools  (p.  82);  he  declared  that  'strategic 
planning  is  worthless  - unless  there  is  first  strategic  vision'  (p.  94).  Naisbitt’s 
views  that  'Followers  create  leaders.  Period.'  (p.  101)  summarizes  the  change  in 
focus.  According  to  the  Saskatchewan  Minister’s  Advisory  Committee  (1985,  p. 
7),  the  impetus  can  come  from  outside  the  school  but  planning  and  action  must 
occur  within.  'School  improvement  is  about  taking  action  at  the  local  level.' 

In  conclusion,  the  school  improvement  literature  has  presented  several 
propositions  and  strategies  for  school  change.  The  consistent  emphasis  on  school 
-level  development  is  incorporated  into  the  section  of  this  case  study  which 
proposes  a prototype  for  school  and  program  evaluation  for  Lethbridge  School 
District  #51. 

Evaluation  Models,  Paradigms,  Criteria,  and  Standards 

In  the  opinion  of  the  researcher,  the  most  significant  debate  related  to  the  best 
model  for  school  evaluation  focuses  on  external  versus  internal  format;  should  the 
evaluation  be  conducted  by  external  evaluators  or  be  done  at  the  school  level  by 
school-based  personnel?  Goodlad  (1984,  p.  31)  addressed  this  issue  with  his 
statement,  'The  approach  having  the  most  promise  is  one  that  will  seek  to  cultivate 
the  capacity  of  schools  to  deal  with  their  own  problems,  to  become  largely  self 
-renewing.' 


Eisner  (1985b,  p.378)  further  supported  the  school-based  approach:  'The 
school  is  the  basic  unit  of  educational  excellence.'  Good  and  Brophy  (1986,  p. 
586)  made  their  contribution  to  the  debate  on  school  evaluation  models  in  stating 
that  one  criterion  for  judging  plans  might  be  the  percentage  of  faculty  involved  in 
devetoping  the  plan  and  the  number  who  accept  it 

Consideration  by  Morgan  (1986)  of  both  the  'top-down'  and  the  'collaborative' 
systems  led  him  to  conclude  that  the  most  successful  systems  of  evaluation  are 
likely  to  be  based  on  a collaborative  approach  since  it  assures  acceptability  which 
may  be  more  crucial  than  validity  and  reliability.  He  stated  a principle  that  program 
evaluation  is  'the  mirror  image  and  complement  of  performance  appraisal'  (p.  61), 
and  that  effective  teaching  is  demonstrated  by  appropriate  and  effective  use  of 
curricular  and  program  materials  and  methods.  As  a note  of  caution,  he  stressed 
that  instead  of  focusing  only  on  classroom  observations,  there  should  be  wider  and 
more  varied  forms  of  appraisal  including  assessment  of  the  diagnosis,  planning, 
choice  of  strategy,  choice  of  materials,  and  methods  of  teaching. 

Common  (1987,  p.  15)  advocated  strongly  against  external  accountability 
because  it  'will  prove  costly  and  contentious,  and  may  lower  teacher  morale.'  She 
argued  further  that  external  evaluations  would  generate  data  which  might  be  used 
very  little,  which  could  deskill  the  teacher,  and  which  could  cause  curriculum 
innovation  to  stop. 

Shaw  (1987-88,  p.  434)  proposed  three  primary  stages  of  school  evaluation: 
self-study  phase,  team-visit/reporting  phase,  and  implementation  phase.  He 
advocated  that  local  school  staff  members  could  effectively  utilize  their  self-study 
instruments,  discussions,  and  activities  to  identify  their  school’s  limitations  and 
develop  a plan  for  school  improvement  Shaw  believed  that  after  a thorough  self 
-study  is  undertaken  and  accomplished,  visiting  team  members  could  serve  as 
external  validators  for  the  work  of  the  local  staff  but  cautioned  that  school  evaluators 
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should  recognize  they  cannot  learn  as  much  during  the  three-day  visit  as  local  staff 
members  already  know  about  their  school  program.  The  study  reported  findings 
comparing  the  evaluative  judgments  of  local  school  staff  members  with  those  of 
visiting  team  members;  13  of  the  19  items  showed  no  significant  difference,  five 
showed  significant  difference,  and  two  items  were  rated  higher  by  the  visiting  team 
(related  to  student  behavior)  while  four  were  rated  higher  by  local  staff  members. 

Novak  (1985)  asserted  that  too  much  stress,  time,  and  money  are  invested  in 
the  formal  preparation  and  visit  involved  in  external  evaluations,  even  though 
agreeing  that  schools  could  benefit  from  some  periodic  outside  review.  This  work 
suggested  broader  involvement  of  the  educational  consumer  in  the  design  and 
operation  of  the  evaluation  process.  He  advocated  shortening  the  external  visitation, 
while  acknowledging  staff  insecurities  if  there  are  insufficient  visiting  subject 
specialists  to  cover  every  discipline. 

An  Australian  model  by  Boud  and  Donovan  (1982)  documented  a set  of 
principles  to  guide  the  practice  of  evaluation:  teachers  are  the  people  who  have  to 
implement  the  changes,  decision-making  is  devolved  to  schools,  internal  evaluation 
replaces  external  evaluation,  and  all  members  of  the  school  participate  in  the 
process.  Baud  and  Donovan  concluded  that  participation  in  the  process  was  just  as 
important  as  the  implementation  of  the  changes,  and  teachers  reported  that  much  of 
the  benefit  was  as  a result  of  their  involvement  in 
the  planning  and  conduct  of  the  evaluation. 

Herman  (1986,  p.  3)  suggested  a model  involving  a 'top-down,  bottom-up' 
approach  which  is  school-district  based  but  oriented  to  meet  school  building, 
classroom,  and  state  needs.  She  noted  the  main  problem  with  existing  top-down 
models  was  that  the  people  at  the  bottom  (teachers  and  local  administrators)  were 
seen  as  data  providers  rather  than  data  users  and  that  paperwork  and  bureaucratic 
burdens  intruded  into,  rather  than  supported,  school  operations  and  improvement 
efforts.  'Bottom-up'  needs  were  not  being  met 

Wilcox  (1989,  p.  188-189)  noted  in  her  British  Columbia  study  of  school  and 
program  evaluation  models  that  'although  their  opinions  about  a program  are 
sought,  rarely  do  stakeholders  have  control  over  the  direction  the  evaluation  takes.' 
Toffier  (1980,  p.  431)  added  arguments  in  favor  of  localizing  decision-making  and 
action  planning  in  order  'to  cure  today’s  decision  logjam.'  He  stated  that  we  need 
to  divide  up  and  reallocate  the  decisions,  sharing  them  more  widely  and  'switching 
the  site  of  decision-making  as  the  problems  themselves  require.' 

In  British  Columbia,  the  Ministry  of  Education  has  developed  procedures  for 
accreditation  which  serve  the  school  and  program  evaluation  function  since 
accreditation  is  defined  in  the  Accreditation  Booklet  for  Secondary  Schools  ( 1983, 
p.  2)  as  'the  outcomes  of  an  internal  and  external  evaluation.'  Internal  evaluation  is 
undertaken  by  the  staff  and  administration  within  the  school  and  is  designed  to 
encourage  and  assist  in  the  improvement  of  the  school  by  its  own  initiative  and 
effort  External  evaluation  is  undertaken  by  an  external  committee  and  is  designed 
to  provide  an  evaluation  in  a broader  frame  of  reference  to  confirm  or  question  the 
internal  evaluation  (p.  2). 

A few  Alberta  school  jurisdictions  place  significant  emphasis  on  a school-based 
model  of  school  and  program  evaluations.  The  Spirit  River  School  Division  #47 
employs  the  B.C.  model  of  internal  and  external  evaluation.  The  Edmonton  Public 
School  Board  employs  a school-based  decision-making  model  which  also  applies  to 
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school  and  program  evaluation.  Edmonton  utilizes  school-initiated  and  school- 
based  evaluation  by  staff  to  assess  school-based  programs.  Such  evaluations  are 
designed,  administered,  and  analyzed  by  school  staff  with  assistance  (on  request)  of 
central  office  support  services.  Teachers  complete  self-awareness  inventories  to 
clarify  their  understanding  of  the  program  and  prepare  them  for  their  discussions 
regarding  their  classroom  practices,  and  to  prepare  them  for  classroom  observations 
and  conference  questions.  At  the  district  level,  they  have  produced  outcomes  and 
expectations  in  language  arts  and  mathematics  and  are  working  on  them  in  other 
subject  areas.  Their  curriculum  is  defined  in  terms  of  intended  outcomes, 
expectations,  and  sample  indicators.  District  achievement  tests  have  been  developed 
based  on  grades  3,  6,  9,  and  12  in  language  arts,  mathematics,  science,  and  social 
studies.  In  addition,  program  evaluations  are  periodically  undertaken  at  the  district 
level  by  curriculum  development  staff. 

In  the  case  of  the  Calgary  Board  of  Education,  the  principal  (with  cooperation  of 
staff)  assesses  all  aspects  of  the  school  rJimate  and  operation,  employing  school  self- 
evaluation,  assisted  by  supervisory  personnel  external  to  the  school.  District-area 
office  personnel  review  the  annual  school  self-evaluation  activities  which  have  been 
put  together  into  a 'School  Profile'  - a framework  to  help  document  and  assess 
present  practices  and  set  priorities  for  future  action.  Their  approach  is  based  upon 
models  from  England,  and  described  in  a document  from  the  City  of  Salford, 
England,  Schools  Looking  at  Themselves  (1983)  and  a publication,  Keeping  the 
School  Under  Review,  by  Casey  and  Malion  (1982).  It  is  grounded  in  the  basic 
assumptions  that  school  evaluation  is  a continuous  process  by  which  the  school  sets 
out  to  improve  itself,  that  schools  differ  within  the  parameters  of  the  provincial 
School  Act  and  board  policies,  that  involvement  of  principal  and  staff  is 
fundamental,  and  that  most  decisions  should  rest  with  them  with  the  school  seeking 
external  advice  and  assistance  as  it  judges  appropriate. 

This  researcher,  after  a thorough  review  of  the  literature,  and  based  on  a career 
in  education  and  evaluation,  concludes  that  the  decision  an  the  debate  related  to  the 
external  versus  internal  evaluation  model  falls  dearly  on  the  side  of  the  school-based 
model.  The  convincing  arguments  of  Boud  and  Donovan  (1982),  Common  (1987), 
Herman  (1986),  TofQer  (1980),  and  Eisner  (1985)  provide  compelling  reasons  for  a 
model  which  involves  more  active  and  professional  participation  of  school-based 
personnel  The  empowerment  of  school-based  teachers  and  administrators,  and  the 
school  as  the  focus  of  action  and  development  offer  the  greatest  potential  for  real 
growth  and  development  of  the  school,  its  staff,  and  its  programs  for  students. 
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Research  Methodology 

a Looking  through  one  eye  never  did  provide  much  depth  of 
field.'  (Eisner,  1981,  p.  9) 

This  research  study,  a meta-evaluation,  utilized  a descriptive-design,  case-study 
approach  since  it  dealt  with  matters  primarily  qualitative  in  nature,  but  also  used 
some  correlational  design  features  as  required  in  analyzing  certain  data.  The  views 
of  Patton  (1980,  p.  40)  who  observed  that  "'researchers  using  qualitative  methods 
strive  to  understand  phenomena  and  situations  as  a whole;  evaluators  using 
qualitative  methods  attempt  to  understand  programs  as  a whole'  would  support  this 
approach. 

Stake  (1978,  p.  5)  observed  that  case  studies  will  often  be  the  preferred  method 
of  research  since  they  may  be  'epistemologically  in  harmony  with  the  reader’s 
experience  and  thus  to  that  person  a natural  basis  for  generalization.'  Patton 
(1980,  p.  304)  presented  a strategy  for  carrying  out  a case-study  and  a format  for  a 
report  outline  (p.  340)  which  has  been  used  in  this  research.  Guba  and  Lincoln 
(1981,  p.  376)  presented  a rationale  for  doing  a case  study  rather  than  the  more 
conventional  technical  report  for  a naturalistic  evaluation. 

Eisner’s  views  (1985a,  p.  358)  support  the  approach  utilized  in  this  Lethbridge 
study  with  his  statement  that  'The  model  of  natural  science  on  which  educational 
research  is  based  is  probably  inappropriate  for  most  of  the  problems  and  aims  of 
teaching,  learning,  and  curriculum  development'  He  advocated  that  educational 
connoisseurship  (the  art  of  appreciation)  and  educational  criticism  be  'not  limited  to 
the  artistic  description  of  events'  but  also  include  'their  interpretation  and  appraisal' 
(p.  155). 

The  literature  was  clear  and  convincing  that  for  the  type  of  naturalistic  research 
being  done  in  this  study,  a qualitative  case-study  design,  supplemented  by  some 
limited  statistical  analysis,  was  the  preferred  mode  for  the  problem  identified 


Sources  of  Data.  Data  Analysis  Procedures 

The  notion  of  'grounded  theory'  as  emerging  from  the  bottom  up  (rather  than 
from  the  top  down),  attributed  to  Glaser  and  Strauss  (1967),  and  reported  in 
Bogdan  and  Biklen  (1982,  p.  29),  stressed  using  the  many  disparate  pieces  of 
collected  evidence  that  are  'interconnected'  The  Lethbridge  study  made  use  of  such 
data  since  large  quantities  of  information  had  been  collected  in  the  past  three  years 
of  school  and  program  evaluations,  using  instruments  designed  by  Lethbridge 
School  District  and  similar  to  those  used  in  other  districts  in  Alberta  and  by  Goodlad 
(1984).  The  Lethbridge  results  were  compared  to  the  findings  of  Goodlad 

In  order  to  control  for  the  possibility  of  researcher  bias  a 'panel  of  experts' 
was  employed  for  inter-subjective  validity  verification.  This  panel  consisted  of  four 
school  principals  who  had  their  schools  evaluated  using  the  Lethbridge  model. 
They  were  asked  for  their  assessment  of  various  aspects  of  the  model  and 
procedures  which  were  followed.  Their  responses  were  compared  to  the 
assessment  made  by  the  researcher. 
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The  data  collected  from  students,  teachers,  and  parents  were  analyzed  to  check 
for  degree  of  association  using  Pearson  product-moment  correlation  (r),  and  also 
compared  with  data  from  the  Goodlad  (1984)  study.  The  data- gathering 
instruments  were  assessed.  The  Lethbridge  model  was  analyzed  and  criticized 
based  on  the  findings  from  the  literature  on  indicators  of  educational  quality, 
effectiveness,  and  school  improvement,  other  school  and  program  evaluation 
research,  and  the  statistical  analyses. 

Responses  from  the  'panel  of  experts',  along  with  other  data  and  information 
from  the  research  study,  were  used  in  a procedure  of  triangulation  to  address  the 
sufficiency  of  the  data  and  to  reach  some  conclusions  on  the  model  currently  being 
employed. 


Findings  (Analysis  and  Meta-Evaluation) 

'The  complete  act  of  evaluation  . . . involves  both  description 
and  judgment (Guba  and  Lincoln,  1981,  p.  380) 

A strong  positive  relationship  was  found  between  the  information  obtained  from 
other  aspects  of  the  school  evaluation  process  and  the  conclusions  reached  based 
upon  the  data  from  the  analyses  of  the  parent,  student,  and  staff  surveys.  The 
grades  awarded  by  each  group  (parents,  students,  and  teachers)  in  Lethbridge  are 
somewhat  higher  than  those  noted  in  the  Goodlad  study.  The  difference  in  ratings 
of  the  Lethbridge  schools  from  those  in  the  Goodlad  study  may  be  accounted  for  by 
the  fact  that  die  studies  occurred  in  different  countries  with  different  social  and 
economic  environments.  The  education  system  in  Canada  has  not  suffered  to  the 
same  degree  from  the  low  levels  of  public  confidence  and  support  as  has  occurred 
in  the  United  States. 

A Pearson  r coefficient  of  correlation  was  computed  on  the  ratings  of  the 
researcher  and  the  Panel  of  Experts.  The  correlation  was  computed  to  be  0.6  which 
can  be  interpreted  to  be  a 'high'  relationship  according  to  Morehouse  and  Stull 
(1975,  p.  198). 

An  analysis  showed  the  areas  of  greatest  agreement  between  the  researcher  and 
the  Panel  of  Experts.  The  researcher  and  the  panel  gave  highly  positive  ratings  to 
statements  covering  several  aspects  of  the  process.  These  included  communication 
with  the  evaluation  team,  opportunity  for  parental  input,  recommendations  which 
were  reasonable  and  accurate,  an  evaluation  report  which  was  presented  in  draft 
form  to  the  school  prior  to  its  finalization  and  release,  and  the  fact  that  the  school 
developed  a plan  to  respond  to  recommendations.  Similar  strongly  positive 
agreement  existed  in  terms  of  the  present  model  being  predominantly  ’top  down', 
and  the  need  for  greater  involvement  of  teachers  and  school-based  administrators. 
Both  the  researcher  and  the  panel  gave  low  ratings  to  the  attention  paid  to  school 
social  inputs  and  to  the  clarity  of  the  evaluators’  role  after  the  report  was  written. 

The  researcher  was  much  less  positive  than  was  the  panel  that  the  input  of 
school  administrators  and  teachers  in  the  present  model  was  adequate.  Close 
agreement  existed  between  the  researcher  and  the  panel  in  a number  of  other  areas. 


Collaborative  Evaluation  Model 


12 


Lethbridge  S.D.*51 


The  usefulness  of  the  evaluation  report  was  rated  fairly  highly  by  each. 

The  approach  to  school  and  program  evaluation  as  used  in  the  Lethbridge 
models  encompasses  virtually  every  one  of  the  indicators  of  effectiveness  presented 
by  Squires  et  al.  (1983)  in  their  questionnaire  for  assessing  school  and  classroom 
effectiveness.  The  Lethbridge  models  appear  to  meet  Goodlad’s  (1984)  assertion 
that  efforts  at  'school  improvement  must  encompass  the  school  as  a system  of 
interacting  parts  each  affecting  the  other"  (p.  31).  The  broadly  based  emphasis  of 
the  models  with  a focus  on  all  aspects  of  the  school  ranging  from  instructional 
programs  to  non-instructional  programs  acknowledges  that  each  part  of  the  school’s 
operation  affects  all  other  aspects. 

The  Lethbridge  model  adequately  addresses  many  of  the  checkpoints  of  the  Key 
Evaluation  Checklist  developed  by  Scriven  (1976)  in  terms  of  description,  clients, 
function,  consumer,  process,  outcomes,  generalizability,  significance,  and 
reporting.  In  the  view  of  the  researcher  and  confirmed  by  the  Panel  of  Experts, 
there  are  some  problems  with  the  delivery  system,  the  standards  by  which  programs 
are  evaluated  (the  lack  of  a clear  set  of  indicators  of  educational  quality),  the 
usefulness  of  the  outcomes  (since  there  are  problems  in  implementing 
recommendations),  the  costs,  and  the  thoroughness  of  the  meta-evaluation. 

The  lack  of  absolute  standards  by  which  programs  can  be  compared  and  assessed 
is  a weakness  of  the  Lethbridge  models.  Another  area  where  the  Lethbridge  models 
are  weak,  according  to  The  'E  Standards'  as  produced  by  North  Central 
Association  Commission  on  Schools  (1987-88),  is  in  teacher  involvement  in 
assessing  the  effectiveness  of  the  program  and  planning  for  its  improvement  The 
present  school  evaluation  procedures  in  Lethbridge  place  teachers  as  recipients  of 
the  process  and  of  the  recommendations  for  change  with  little  real  and  meaningful 
involvement  in  determining  the  nature  of  changes  desirable.  Teacher  involvement 
comes  into  place  in  the  present  models  only  after  the  external  evaluators  have 
determined  what  changes  should  be  made.  The  present  school  evaluation  models 
include  procedures  to  check  that  the  characteristics  common  to  effective  schools  are 
in  place. 

The  lack  of  clearly  established  and  accepted  indicators  of  educational 
effectiveness  or  quality  is  one  of  the  major  weaknesses  of  the  Lethbridge  approach 
to  school  evaluation.  Although  many  of  the  qualitative  and  quantitative  indicators 
may  be  inferred  in  the  Lethbridge  models,  they  are  not  specifically  identified  and 
looked  for  as  part  of  the  evaluation.  Little  emphasis  is  placed  on  interpretive 
indicators  of  context,  input,  and  process,  or  on  outcome 

indicators  of  a cognitive,  affective,  or  behavioral  nature.  A set  of  standards  or 
bases  for  comparison  is  absent  The  result  is  that  judgments  are  made  about 
program  quality  and  effectiveness  without  the  strength  of  a set  of  agreed-upon 
standards. 

The  literature  on  school  improvement  leads  to  some  criticisms  of  the  school 
evaluation  model  in  question.  The  concerns  of  Leithwood  and  Fullan  (1984)  have 
not  been  addressed  adequately.  They  believed  that  successful  change  involves 
pressure  gradually  acquired  through  interaction  with  peers  and  other  leaders,  not  an 
imposed  pressure  mandated  by  authority.  The  present  evaluation  models  place 
emphasis  for  change  from  the  pressure  brought  about  by  the  evaluation  report 
instead  of  from  interaction  with  peers.  Similarity,  the  collaborative  planning  as 
espoused  by  Patterson,  Pur  key,  and  Parker  (1986)  is  limited  and  restricted  by  the 
present  models.  The  current  LSD  #5 1 approach  does  not  utilize  the  suggestions  of 
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Landon  and  Shirer  (1986)  in  the  Wisconsin  School  Evaluation  Plan  to  have  the 
school  conduct  a seif -evaluation  which  is  then  audited  by  an  outside  team. 

Wood,  Freeland,  and  Szabo  (1985)  stressed  a focus  on  staff  development 
instead  of  the  traditional  emphasis  on  curriculum  development,  and  for  planning  that 
is  proactive,  long  range,  and  systematic.  The  Lethbridge  evaluation  model  is  not 
consistent  with  the  suggestions  of  these  and  several  other  researchers  in  the  area  of 
school  improvement 


Summary,  Conclusions,  and  Recommendations  - 
Revising  the  Model 

* Trends  are  bottom-up,  fads  top-down .*  (Naisbitt,  1982,  p.3) 

The  findings  of  the  case  study  are  that  the  Lethbridge  School  District  #5 1 model 
for  school  and  program  evaluations  has  generally  been  perceived  to  be  moderately 
effective  and  has  led  to  significant  efforts  at  school  and  program  improvement  The 
major  strengths  of  the  model  are  the  involvement  of  large  numbers  of  people  in 
providing  data  on  which  evaluative  judgments  can  be  made.  All  stakeholders  have 
adequate  opportunity  to  make  input  and  have  their  views  known.  Another 
perceived  strength  (which  is  also  a weakness)  is  the  use  of  a large  team  of  external 
evaluators  with  a strong  range  of  program  and  administrative  strengths.  The 
credibility  of  the  evaluation  team  is  strong.  The  countering  weakness  is  that  with 
such  a large  and  strong  team,  the  model  has  become  heavily  'top-down'  with  little 
real  opportunity  for  meaningful  participation  and  professional  growth  by  those  who 
are  the  recipients  of  the  evaluation  - the  school  staff  and  administrative  team.  The 
lack  of  teacher  involvement  and  interaction  with  peers  in  assessing  the  effectiveness 
of  their  own  programs  and  then  planning  for  improvement  is  a fundamental 
concern. 

The  model  was  judged  to  be  very  strong  in  its  examination  of  a broad  range  of 
student  behaviors,  teacher  behaviors,  school  climate,  student  achievement,  and 
expectations  for  success.  The  focus  on  all  aspects  of  the  school  encompassing  both 
instructional  and  non-instructional  programs  was  positively  evaluated.  The 
multidimensional  nature  of  the  model  is  a strength.  Some  uncertainty  exists  as  to 
whether  the  information  gained  from  the  model  is  of  sufficient  value  in  terms  of 
professional  growth  and  improvement  to  warrant  the  cost  of  human  resource  time 
and  energy. 

Other  major  concerns  with  the  model  are  the  lack  of  a clearly  established  and 
accepted  set  of  standards  or  indicators  of  quality  and  the  problems  in  implementing 
the  recommendations  when  members  of  the  district  central  staff  are  heavily 
committed  to  carrying  out  other  evaluations.  The  thoroughness  of  the  'evaluation 
of  the  evaluation,'  whereby  the  evaluation  process  and  model  is  regularly  evaluated, 
is  another  weakness  of  the  model. 

This  case  study  found  that  the  Lethbridge  model  satisfied  many  of  the  criteria  of 
effectiveness,  quality,  and  improvement  in  terms  of  receiving  input  from  all 
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stakeholders,  utilizing  an  evaluation  team  with  expertise  and  credibility,  and  using  a 
multidimensional  focus  to  examine  a broad  range  of  both  instructional  and  nan- 
instructional  aspects  of  the  school.  The  model  was  judged  by  die  researcher  to  have 
limitations  in  that  it  was  'top-down'  with  little  opportunity  for  meaningful 
participation  and  professional  growth  by  the  school  staff  and  administration.  There 
are  strong  doubts  as  to  whether  the  information  gained  from  the  model  is  of 
sufficient  value  related  to  professional  growth  and  improvement  to  warrant  the 
heavy  costs  of  human  resource  time  and  energy.  A major  weakness  of  the  model  is 
the  lack  of  clearly  established  and  accepted  sets  of  standards  or  indicators  of  quality. 
Concerns  with  the  ability  of  schools  to  implement  the  recommendations  leads  to 
serious  questions  about  whether  significant  improvements  will  come  about 

This  case-study  determined  that  the  procedures  and  instruments  used  had  strong 
face,  content,  and  construct  validity.  No  attempt  was  made  to  demonstrate  external 
validity  since  the  purpose  was  not  to  generalize  the  conclusions  reached  at  one 
school  to  another  one.  Some  doubts  about  reliability  and  validity  were  raised  since 
the  items  were  not  field  tested  and  some  items  were  changed  from  school  to  school 
with  loss  of  comparability  of  certain  results.  Although  reliability  (internal 
consistency  or  stability)  was  never  calculated,  the  instruments  appear  to  be 
consistent  as  indicated  by  the  high  correlations  which  were  found. 

The  literature  was  clear  that  there  are  certain  criteria  and  characteristics  which 
could  be  incorporated  into  the  Lethbridge  school  evaluation  procedures  to  increase 
effectiveness  and  efficiency: 

1.  The  new  model  should  move  away  from  the  heavy  emphasis  on  a 'top- 
down'  approach  to  encourage  and  empower  school  staff  and  administrators  to 
be  actively  involved  and  interacting  in  Msgaamg  the  effectiveness  of  their  own 
programs  and  planning  for  their  improvement  A model  involving  a better 
balance  between  internal  and  external  evaluation  should  be  considered.  School 
-based  subject-area  evaluation  by  internal  sub-committees,  followed  up  by  a 
review  by  the  external  team  to  confirm,  question  or  add  new  factors  to  the 
assessment  is  recommended. 

2.  The  development  of  a clear  set  of  standards  or  indicators  of  educational 
quality  would  be  a desirable  step  in  moving  the  evaluation  model  onto  more 
objective  ground,  and  would  allow,  perhaps  even  demand,  much  more 
professional  involvement  of  school  level  personnel.  Indicators  should  be 
developed  by  school-level  committees  of  teachers  and  administrators,  and 
shared  with  and  revised  as  necessary  by  other  schools.  The  process  of 
developing  educational  quality  indicators  is  an  evolutionary  one;  the  result,  after 
a period  of  approximately  three  years,  should  be  a district-wide  set  of  indicators 
which  have  been  teacher-developed  and  validated.  The  indicators  may  vary 
somewhat  between  schools  and  particularly  between  levels  or  divisions  to  take 
into  account  their  unique  features.  Indicators  should  consider  the  school  social 
inputs  such  as  the  student  body  composition,  and  the  school  social  structure  and 
climate,  and  should  focus  on  student  outcomes  in  the  cognitive,  affective,  and 
behavioral  domains  with  reference  to  context,  inputs,  process,  comparators,  and 
standards.  The  work  of  McEwen  and  Zatko  (1989),  Shedlin  (1986),  and  the 
Alberta  (1989b)  and  British  Columbia  (1986)  papers  on  quality  indicators  have 
produced  numerous  examples  of  quality  indicators  in  the  various  domains. 

3 . Evaluation  of  the  teaching  staff  and  evaluation  of  the  principal  need  not  be  a 
part  of  the  school  and  program  evaluation  process.  The  principal  should 
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assume  full  responsibility  (with  assistance  and  delegation  as  required)  for 
teacher  evaluation,  while  principal  evaluation  should  be  carried  out  under  the 
direction  of  the  district  superintendent 

4.  The  Lethbridge  model,  to  become  more  effective,  should  improve  its 
approaches  to  and  emphasis  on  implementation;  the  central-  office  resources 
and  professional  expertise  should  not  be  so  heavily  involved  with  planning 
and  carrying  out  other  evaluations  that  they  cannot  be  available  for  the  most 
important  of  all  steps  in  die  evaluation  process  which  is  implementation  of 
recommendations. 

5.  More  deliberate  and  thorough  efforts  must  be  put  into  meta-evaluation 
whereby  the  model  and  processes  would  be  evaluated  by  all  stakeholders  in  the 
process  and  coordinated  by  a district  school-evaluation  coordinating  committee. 
All  who  were  involved  in  the  process  of  evaluation  and  all  who  have  a stake  in 
the  school  should  have  input  into  the  meta-evaluation.  The  board  of  education, 
as  recipients  of  the  evaluation  report,  should  also  have  significant  input  into  the 
meta-  evaluation  in  terms  of  whether  it  produced  necessary  and  useful 
information.  Related  information  on  the  value  and  effectiveness  of  the 
educational  quality  indicators  developed  by  each  school  will  come  from  the 
judgements  of  the  teachers  in  other  schools  who  will  make  use  of  these  (or  a 
modification  of  these)  indicators  in  their  own  school  and  program  evaluations. 
This  will  assist  in  refining  the  process  of  developing  quality  indicators. 

6.  School  and  program  evaluation  in  LSD  #51  should  be  an  ongoing  process 
and  not  simply  an  event  which  takes  place  once  every  five  to  seven  years.  The 
new  model  should  assure  that  program  and  school  evaluation  becomes  a natural 
and  vital  part  of  the  curriculum  and  instruction  cycle.  Each  school  in  LSD  #51 
should  develop  a school-  and  program-evaluation  component  as  a part  of  its 
operational  plains. 

7.  In  the  long  term,  school  and  program  evaluation  should  be  happening 
simultaneously  in  all  schools  of  LSD  #5 1 ; the  only  scheduling  required  from  the 
district  should  be  for  the  external  evaluation  component 
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Figure  1 


Proposed  Collaborative  Mode!  for  School  Evaluation 
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Proposed  Collaborative  Model  of  School  Evaluation 

Based  on  the  findings  of  the  case-study,  a new  model  for  school  and  program 
evaluation  is  proposed.  The  diagram  (see  Figure  1)  contains  a proposed 
collaborative  model  for  the  evaluation  of  schools  and  includes  all  the  criteria  and 
characteristics  outlined  earlier  in  this  chapter  to  improve  the  effectiveness  and 
efficiency  of  the  existing  Lethbridge  model.  This  collaborative  model  places  much 
greater  responsibility  and  control  in  the  hands  of  school-based  administrators  and 
staff.  They  become  major  participants  and  decision-makers  in  program  and  school 
evaluation,  in  identifying  areas  of  program  and  professional  growth,  and  in 
bringing  about  change. 

The  model  rests  upon  certain  basic  assumptions  grounded  in  the  literature  on 
school  effectiveness  and  school  improvement.  The  school  is  the  primary  unit  of 
decision-making  (Smith  and  Purkey,  1985).  If  changes  are  to  occur,  they  require 
ownership  that  comes  from  the  opportunity  to  participate  in  defining  change,  and 
the  flexibility  to  adapt  it  to  individual  circumstances;  change  does  not  come  from 
externally  imposed  procedures  (Fullan,  1982).  School  evaluation  should  be  a 
continuous  process.  Schools  want  to  identify  areas  requiring  improvement  and  will 
actively  work  toward  this  end  if  the  conditions  are  right  Schools  differ 
within  the  parameters  of  school-district  policies  and  provincial  legislation;  these 
differences  must  be  recognized 

and  acknowledged  in  the  evaluation  and  improvement  processes. 

Empowering  school-based  staff  and  administration  is  fundamental  to  reviewing 
and  improving  the  school;  an  empowered  school-based  staff,  with  assistance  from 
external  expertise  (as  required),  will  make  strong  professional  decisions  regarding 
their  own  program  improvement  and  professional  development  The  entire  school, 
the  environment  in  which  students  learn,  and  ultimately  the  students  themselves, 
will  be  the  benefactors.  Glickman  (1989,  p.  8)  supported  this  view  of  school 
improvement  He  stated  that  'schools  will  not  improve  until  those  people  closest  to 
the  students  - the  teachers  - are  given  the  choice  and  responsibility  to  make 
collective  and  informed  decisions.'  He  elaborated  by  stating  that  'supervision  must 
shift  decision  making  about  instruction  from  external  authority  to  internal  control.' 
Speaking  of  teachers,  he  stressed  that  'without  choice  and  responsibility,  they  will 
comply,  subvert,  or  flee;  and  motivation,  growth,  and  collective  purpose  will 
remain  absent' 

The  two  areas  of  literature,  school  improvement  and  school-based  management, 
are  brought  together  by  David  (1989,  p.  45).  She  asserted  that  'school-based 
management  is  rapidly  becoming  the  centerpiece  of  the  current  wave  of  reform.'  If 
school  and  program  evaluations  are  for  the  purpose  of  instructional  improvement  or 
reform,  then  the  conclusions  of  David  (1989),  Glickman  (1989),  Smith  and  Purkey 
(1985),  Fullan  (1982),  and  others  can  not  be  ignored.  This  researcher  is 
convinced,  based  on  the  literature  and  on  personal  experience  in  the  field,  that  the 
school-based  empowerment  approach  is  the  one  most  likely  to  bring  about  the 
desired  staff,  program,  and  school  development 
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The  Lethbridge  EQI  Project  - Future  Directions 

The  present  model  of  program  and  school  evaluation  as  used  in  Lethbridge 
School  District  #5 1 was  evaluated  according  to  recent  research  findings  on  school 
effectiveness,  quality  indicators,  and  school  improvement,  and  according  to  theories 
and  practices  of  sound  evaluations.  It  was  found  to  be  adequate  in  many  of  the 
criteria  of  effectiveness,  quality,  and  improvement,  but  to  be  insufficient  in  a 
number  of  other  important  ways  identified  from  theories  and  practices.  The 
proposed  collaborative  model  adds  a number  of  features  which  should  empower 
school-based  administrators  and  staff,  strengthen  the  evaluation  processes,  and 
increase  school  improvement  and  professional  development  and  growth.  The 
proposed  model  will  be  implemented  on  an  experimental  basis  in  Lethbridge  School 
District  #5 1 , evaluated,  and  modified  as  necessary  to  strive  toward  the  achievement 
of  sound  program  and  school  evaluations  and  the  ultimate  goal  of  improved 
educational  experiences  for  students. 

Based  on  the  literature  and  on  the  results  of  the  case-study,  a new  approach  to 
school  and  program  evaluation  is  underway  in  Lethbridge  School  District  #5 1 . The 
Lethbridge  EQI  project  involves  implementation  of  a collaborative  model  of 
program  and  school  evaluation  with  emphasis  on  the  development  of  educational 
quality  indicators.  Key  components  of  the  development  of  educational  quality 
indicators  are  the  collaboration  between  school-based  and  district-based  staff,  and 
the  resulting  determination  of  comparators,  standards,  and  targets  for  all  programs. 
These  educational  quality  indicators  will  focus  on  student  outcomes  in  the  cognitive, 
affective,  and  behavioral  domains,  and  will  be  developed  within  an  interpretive 
framework  which  places  emphasis  on  context  (the  basis  for  interpreting  the 
variations  among  different  groups),  inputs  (the  number  and  type  of  resources 
available  and  allocated)  and  process  (the  actions  or  operations  which  result  in  an 
outcome).  Data  from  surveys  of  students,  teachers,  parents,  and  the  public  will  be 
analyzed  and  quality  indicators  established  in  terms  of  comparators,  standards,  and 
targets.  The  result  will  be  each  school  evaluating  its  own  educational  quality  and 
effectiveness  in  a professional  manner  which  serves  to  empower  school-based  staff 
and  contribute  to  meaningful  and  effective  school  growth  and  improvement 
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School  System  Review: 
A Comprehensive  Process 


This  paper  describes  the  cooperative  development  of  a comprehensive  school 
system  review  process  based  on  student  outcomes  as  well  as  the  inputs  and 
processes  that  contribute  to  those  outcomes.  The  focus  is  on  developing  an 
assessment  process  for  the  whole  school  system  to  meet  two  needs:  the  demand  by 
the  public  for  accountability,  and  a desire  by  the  school  system  personnel  to 
demonstrate  their  ability  to  improve  educational  programs. 

Introduction 

The  County  of  Lacombe  and  the  Rocky  Mountain  School  Division,  two  rural 
school  jurisdictions,  have  had  an  interest  in  the  development  of  an  educational 
quality  indicators  system  for  some  time.  Because  of  their  concern  for 
accountability  and  school  improvement,  both  jurisdictions  have  since  1985 
conducted  student,  parent,  public  and  employee  surveys  on  their  satisfaction  with 
the  school  systems . As  a result,  when  the  opportunity  arose  to  become  involved  in 
the  provincial  initiative  on  educational  quality  indicators , the  two  jurisdictions 
submitted  a successful  project  proposal  to  develop  a more  comprehensive  review 
process  based  on  a quality  indicators  system. 


The  project  was  approved  in  April,  1989  and  will  consist  of  three  phases  over  three 
years:  development,  field-testing  and  implementation.  The  first  section  of  the  paper 
presents  the  background  to  the  development  of  the  comprehensive  review  process, 
including  the  purpose,  outcomes  expected  and  related  literature.  The  second 
section  describes  the  conceptual  framework  and  the  development  of  the  components 
of  the  review  process.  The  final  section  describes  future  developments  and  makes 
some  observations  about  the  project 


Purpose  and  Outcomes  of  the  Study 

The  purpose  of  the  project  is  to  develop  a comprehensive  school  system  review 
process  which  can  assess  the  relative  health  of  educational  systems  by  focussing  on 
both  student  outcomes,  and  the  inputs  and  processes  that  contribute  to  those 
outcomes. 


The  development  of  this  school  system  review  process  will  result  in:  a clear 
statement  of  goals  of  student  learning;  a system  of  quality  indicators  for  all  goals  of 
student  learning;  an  interpretative  process  that  will  relate  context,  input  and  process 
variables  to  the  outcomes  achieved,  and  a plan  for  enhancing  the  strengths  and 
improving  areas  of  weakness.  All  of  the  components  of  the  review  process  will  be 
developed  through  the  full  involvement  of  the  educational  partners,  thereby 
enhancing  support  for  the  project 
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Related  Literature 


As  public  demands  for  accountability  and  improved  educational  productivity  have 
grown,  the  interest  in  developing  educational  quality  indicator  systems  to  assess 
the  quality  of  schools,  districts  and  provincial  and  national  educational  systems  has 
increased.  The  result  has  been  a proliferation  of  articles  and  papers  on  not  only 
developing  appropriate  indicators  but  also  providing  the  processes  on  data 
collection,  analysis  and  reporting  of  findings. 


This  literature  review  has  been  completed  in  two  general  sections:  articles  and 
papers  about  the  development  and  implementation  of  quality  indicator  systems,  and 
those  papers  that  analyze  the  inputs  and  processes  that  affect  student  learning.  A 
cross-section  of  papers  and  studies  includes:  studies  on  input,  process  and 
outcome  indicators,  data-collection,  analysis  and  reporting.  The  second  part  of  the 
review  provides  a number  of  meta-analyses  of  several  hundred  studies  on  factors 
that  affect  student  learning  in  addition  to  specific  studies. 


Education  quality  indicators  are  generally  defined  in  the  literature  as  statistical 
measures  that  assess  or  are  related  to  a desired  outcome  of  the  educational  system, 
or  describe  a central  feature  or  features  of  the  system  (Oakes,  1986).  They  are 
usually  expressed  as  a number  which  can  be  compared  longitudinally  with  system 
results,  or  externally  with  other  jurisdictions’  results.  Indicator  systems  are  defined 
as  organized  sets  of  vital  signs  from  which  it  is  possible  to  make  judgements  about 
the  health  of  an  educational  system  (Goertz,  1989;  Oakes,  1986). 


The  literature  suggests  that  effective  indicator  systems  should  include  data  on 
educational  inputs,  processes  and  outcomes  (Fraser,  Walberg,  Welch,  and  Hattie, 
1987;  Haertel  and  Katzenmeyer,  1989;  Walberg,  1983.)  Inputs  include  variables 
over  which  schools  can  exert  some  influence  such  as  resources  and  teacher 
characteristics,  as  well  as  contextual  factors  over  which  jurisdictions  have  no 
control,  such  as  student  mobility  and  socio-economic  status  . Processes  are  defined 
as  the  set  of  activities  that  expose  students  to  the  opportunities  to  learn  (Fraser  et  al, 
1987)  and  include  such  variables  as  classroom  and  teaching  practices,  and 
curriculum  content  and  quality.  Outcomes  refer  to  the  results  achieved  including 
student  achievement,  participation  rates  and  attitudes. 


The  literature  identifies  a number  of  characteristics  of  effective  quality  indicators. 
Indicators  should  : measure  a central  feature  of  the  system  (Office  of  Education 
Research  and  Improvement,  1988),  provide  a benchmark  for  determining  progress 
or  regression  over  time,  and  be  understood  by  all  concerned  with  education  (U.S. 
Department  of  Education,  1985).  Quality  indicator  systems  should  be  based  on  a 
local  interpretation  of  provincial  goals  of  schooling,  and  limited  to  a small  number 
of  measures  to  avoid  interference  with  instruction  (David,  1986).  These  systems 
should  also  include:  logical  clusters  of  indicators  and  a variety  of  data  collection 
sources  (Goertz,  1989),  involve  school  and  district  staff  in  developing  the  indicator 
system  (David,  1986),  and  to  be  useful  for  improving  education,  input  and  process 
variables  must  be  related  to  outcomes  (Haertel  and  Katzenmeyer,  1989;  Goertz, 
1989). 
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The  literature  suggests  three  general  methodologies  for  collection  of  information 
(Buccino,  1989;  Overgaard,  1988):  review  of  documents  such  as  annual  reports, 
census  data  and  records;  testing  which  includes  provincial  assessments,  diploma 
examinations,  standardized  tests  and  locally  developed  tests,  and  surveys  including 
opinion  questionnaires,  interviews  and  polls. 


To  fulfill  the  demand  for  accountability  and  to  demonstrate  the  effectiveness  of  the 
school  systems,  the  literature  suggests  that  care  must  be  taken  to  design  and  publish 
appropriate  reports.  Goertz  (1989)  suggests  a number  of  factors  that  should  be 
addressed  in  designing  these  reports: 


• primary  purpose 

• primary  approach  (descriptive/analytical  or  both) 

• intended  audiences 

• frequency  of  publication 

• level  of  reporting  units  ( district  / school  reporting). 

From  the  reports  on  quality  indicator  system  projects,  it  is  evident  that  effective 
presentation  of  the  data  includes  a combination  of  methods;  a comprehensive 
printed  public  report;  presentations  to  community  groups  by  trustees  and 
administrators;  newspaper,  radio  and  television  clips  summarizing  the  results  of  the 
review;  and  school  newsletters  to  parents. 


The  literature  review  on  inputs  and  processes  that  affect  student  learning  indicates 
three  categories  of  findings:  student  aptitude  related  variables;  instructional  process 
variables  (including  instructional  methodology  and  organization,  and  school  and 
system  effectiveness);  and  context  related  variables  such  as  home  environment,  peer 
group  and  parental  attitude  toward  education. 


The  literature  relative  to  student  aptitude  suggests  that  intellectual  ability, 
developmental  stage  and  motivation  (Fraser  et  al,  1987)  are  strong  indicators  of 
student  success.  As  well,  many  instructional  methodology  and  organization 
variables  were  found  to  positively  influence  student  achievement  including: 
reinforcement  and  cooperative  learning  (Fraser  et  al,  1987);  time  on  task  and 
individualized  instruction  (Walberg,1983);  and  wait  time  (Wise  and  Okey,  1983). 
Additionally,  the  school  effectiveness  literature  identifies  a number  of  other  factors 
which  positively  affect  outcomes  including:  teacher  inservice  (Sweitzer  and 
Anderson,  1983),  focus  on  learning  (Chandler,  1988),  varied  instructional  methods 
that  are  responsive  to  student  learning  needs  (Chandler,  1988),  high  expectations 
(National  Regional  Educational  Laboratory,  1985)  and  a positive  working 
environment  (Mortimore,  1985).  A number  of  contextual  factors  were  found  to 
correlate  positively  with  student  learning  including:  the  family  socio-economic 
status  (Heyneman  and  Loxley,  1983);  peer  group  attitude  toward  achievement; 
home  environmental  factors  such  as  parental  support  of  schooling  (Fraser  et  al. 
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1987);  and  student,  peer  and  parental  support  of  traditional  values  such  as  the  work 
ethic  (Hanson  and  Ginsburg,  1985). 


In  summary,  the  literature  on  the  development  and  implementation  of  a quality 
indicator  system  combined  with  a review  of  the  research  on  inputs  and  processes 
that  influence  student  outcomes  provides  the  necessary  information  to  develop  a 
conceptual  framework  for  the  project  which  follows  in  the  next  section. 


School  System  Review:  Development  of  the  Components 

The  conceptual  framework,  predicated  on  the  sources  that  have  previously  been 
cited,  provides  overall  guidance  to  the  development  of  the  components  of  the 
system  review  process.  Besides  describing  the  conceptual  framework,  this  section 
outlines  the  process  of  delineating  the  gods  of  learning  and  identifying  indicators 
and  measures.  In  addition,  the  indicators,  measures  and  methodology  and  the 
interpretative  framework  and  process  are  described. 


Conceptual  Framework 

The  school  system  review  process  is  based  on  an  educational  quality  indicator 
system  consisting  of  a set  of  indicators  and  related  methodology  to  assess  the 
performance  of  the  school  system  on  the  achievement  of  its  goals.  The  indicators 
should  provide  information  to  assist  in  ’’assessing  the  quality  of  educational 
programs  and  the  delivery  system  by  focussing  on  student  outcomes"  (McEwen 
and  Zatko,  1989,  p.l). 


Specifically,  an  educational  quality  indicator  system  should  include: 

• a local  interpretation  of  provincial  goals  of  schooling  and  a range  of 
cognitive,  affective  and  behavioral  goals; 

• multiple  indicators  or  clusters  of  indicators,  to  increase  interpretative  power, 
but  be  limited  to  a reasonable  number  of  measures  for  which  valid  and 
useful  information  can  be  gathered; 

• a focus  on  indicators  that  are  enduring,  easily  understood,  feasibly 
measured  and  generally  accepted  as  valid  statistics; 

• a point  of  comparison  such  as  a larger  group  or  a previous  result,  and 
include  input  and  process  variables  to  assist  in  interpreting  outcomes; 

• information  that  can  be  readily  used  for  planning  improvement  and  policy 
decisions; 

• all  educational  partners  in  the  specification  of  goals  and  indicators, 
collection  of  information,  interpretation  of  results  and  development  of 
improvement  plans; 
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a reporting  system  that  is  accurate  and  timely  with  appropriate  reporting  for 
all  administrative  units. 


The  conceptual  framework  for  the  school  system  review  process  consists  of  five 
distinct  but  interrelated  components: 

1 . a plan  for  involvement  of  partners  in  the  specification  of  goals,  identification 
of  indicators  and  comparators,  interpretation  of  results  and  the  development 
of  a follow-up  improvement  plan; 

2.  a comprehensive  statement  of  the  goals  of  student  learning  including 
cognitive,  affective  and  behavioral  goals  with  a delineation  of  goals  that  are 
the  primary  responsibility  of  the  school  and  those  that  are  shared  with  the 
family  and  community; 

3 . a system  of  indicators  that  measures  all  of  the  goals  of  student  learning; 

4.  an  interpretative  framework  including  standards  as  well  as  context,  input 
and  process  variables  that  affect  achievement,  and 

5 . a plan  for  the  future  that  identifies  strategies  and  related  action  plans  to 
enhance  student  performance  in  areas  of  strength  and  to  improve 
performance  in  areas  req  :iring  attention. 

The  educational  partners  will  be  involved  in  the  development  of  all  components  of 
the  school  system  review  process:  the  specification  of  goals,  identification  of 
indicators  and  standards,  interpretation  of  results  and  the  development  of  an 
improvement  plan.  Local  committees  in  each  jurisdiction,representing  a broad 
cross-section  of  the  educational  community, will  be  involved  in  the  developing  the 
review  process,  in  sanctioning  directions  for  the  project  and  in  communicating  the 
project  progress  to  their  groups.  Additionally,  direct  involvement  of  parents, 
teachers  and  students  in  development  of  some  components  of  the  project  will  occur. 


The  development  of  a comprehensive  statement  of  goals  of  student  learning  by  the 
school  jurisdictions  is  a necessary  starting  point  for  the  system  review.  Clearly, 
evaluation  must  be  based  upon  specific  results  that  the  jurisdictions  wish  to  achieve 
taking  into  account  provincially  mandated  goals  and  curricula.  Accordingly,  goals 
of  schooling  will  be  developed  utilizing  a process  that  provides  the  educational 
partners  with  the  opportunity  to  meaningfully  interpret  provincial  goals  within  the 
local  context  The  comprehensive  goals  statement  will  include  cognitive,  affective 
and  behavioral  goals  of  student  learning  and  delineate  which  goals  are  the  primary 
responsibility  of  the  school  and  which  are  a shared  responsibility  between  the 
school,  family  and  community.  This  distinction  is  necessary  to  meaningfully 
interpret  and  understand  the  outcomes  achieved,  as  well  as  to  assign  responsibility 
for  improvement  plans. 


Once  the  goals  of  student  learning  are  established  they  will  become  the  focus  for  the 
development  of  a system  of  quality  indicators  designed  to  measure  the  degree  to 
which  all  of  the  agreed  upon  goals  are  being  achieved.  The  system  of  indicators 
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will  be  developed  through  direct  and  full  involvement  of  parents  and  staff  and 
should  be  guided  by  the  qualities  previously  identified  in  the  introduction  to  the 
conceptual  framework.  Through  this  process,  acceptance  of  the  indicators  and 
measures  will  be  developed 


In  addition  to  clearly  articulating  system  goals  and  appropriate  indicators  or  clusters 
of  indicators,  to  measure  the  degree  to  which  established  goals  are  being  achieved, 
the  indicator  system  will  include  an  interpretative  framework  including  comparators 
as  well  as  context,  input  and  process  variables  that  affect  student  achievement. 
Through  this  interpretative  process,  educational  outcomes  can  be  fully  understood 
when  they  are  analyzed  in  relationship  to  the  local  context  - which  tends  to  be 
relatively  fixed-  and  inputs  and  processes,  which  can  be  modified  to  achieve  desired 
results. 


Once  the  indicator  system  is  established  and  the  student  achievement  of  goals  is 
determined  through  measurement  and  interpretation,  the  local  steering  committee 
will  develop  strategies  and  related  action  plans  to  enhance  student  performance  in 
areas  of  strength  and  to  improve  performance  on  goals  requiring  attention.  A 
modified  strategic  planning  process  (Cook,  1988)  will  include: 

• reviewing  the  strategic  policies  of  district; 

• completing  internal  and  external  environment  assessments; 

• reviewing  the  results  achieved  by  the  district  in  relation  to  the  established 
goals; 

• identifying  critical  issues  and  concerns; 

• establishing  specific  improvement  goals; 

• developing  strategies  to  achieve  the  goals,  and 

• reviewing  the  education  quality  indicator  system  to  ensure  that  it  will 
provide  required  data  to  monitor  progress  and  performance. 


Developing  the  Components 

The  following  section  indicates  the  process  of  developing  the  specific  components 
of  the  review  process  which  include  delineating  the  goals  of  student  learning  and 
identifying  indicators,  measures  and  comparators.  The  section  concludes  with  a 
description  of  the  interpretative  framework  and  process. 


Comprehensive  Statement  of  Goals:  The  project  steering  committee  in  the  County 
of  Lacombe  and  the  Rocky  Mountain  School  Division  met  for  two  full  days  to 
delineate  a comprehensive  statement  of  goals  of  student  learning.  In  developing 
this  statement  of  goals,  the  committee  considered  a number  of  goal  statements  from 
the  Department  of  Education  including:  the  goals  of  schooling, the  shared  goals  of 


A Comprehensive  Process 


6 


County  of  Lacombe, 
Rocky  Mountain  School  Division 


education,  the  goals  of  elementary  and  secondary  education,  the  statement  of 
desirable  personal  characteristics  of  students,  and  essential  concepts,  skills  and 
attitudes  of  senior  high  students. 


To  assist  the  committees  in  dealing  with  these  various  statements,  the  project 
coordinator  regrouped  all  of  statements  into  four  categories  of  goalsicognitive, 
social,  person^  and  vocational  (areas  of  overlap  were  noted).  In  addition  the 
committees  were  provided  with  guidelines  for  developing  and  assessing  goal 
statements: 

• any  goal  must  include  the  intents  of  provincial  goals; 

• local  goals  can  be  added  to  the  provincial  statements; 

• the  goals  of  schooling  as  well  as  the  goals  of  education  must  be  evaluated; 

• each  goal  must  be  mutually  exclusive; 

• any  comprehensive  statement  of  goals  must  include:  cognitive,  behavioral 
and  affective  goals; 

• all  goals  must  be  measurable. 

The  group  processes  that  were  organized  resulted  in  gaining  consensus  on  fourteen 
goals  in  the  County  of  Lacombe  and  fifteen  in  Rocky  Mountain  School  Division. 


Following  this  developmental  process  the  goals  were  communicated  to  various 
groups  by  the  steering  committee  members  as  well  as  through  Superintendents' 
newsletters.  Although  no  feedback  was  received  at  this  point,  the  goal  statements 
were  refined  in  the  next  stage  of  the  process  when  indicators  and  measures  where 
identified.  Participants  at  this  stage  found  overlap  between  goals  and  concluded 
that  other  goals  were  not  measurable.  As  a result  the  statement  of  goals,  which  is 
included  in  the  Appendix,  has  been  modified  and  now  contains  eight  goals  in  each 
school  system. 


The  significance  of  this  exercise  of  developing  a goal  statement  is  that  these 
jurisdictions  now  have  a statement  of  goals  of  student  learning  with  a local  flavor 
that  all  educational  partners  understand  and  accept  as  the  basis  for  educational 
programs  and  the  starting  point  for  identifying  indicators  essential  to  the  review 
process. 


The  Process  for  Identifying  Indicators  and  Measures:  The  purpose  of  this 
development  was  to  identify  outcome  indicators  and  existing  or  required  measures 
to  assess  student  achievement  on  the  previously  specified  goals  of  student  learning. 
Meetings  involving  parents  and  teachers  in  the  County  of  Lacombe  and  Rocky 
Mountain  School  Division  were  held  to  identify  educational  quality  indicators, 
existing  or  required  measures  and  comparators.  Workshop  sessions  were  held  in 
each  school  in  the  two  jurisdictions  at  which  time  the  project  was  outlined. 
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indicators  and  measures  were  identified,  and  recommendations  for  other  necessary 
measurement  instruments  were  considered. 


At  the  meetings  after  an  initial  presentation  to  explain  the  project,  answer  questions 
and  outline  the  task,  individual  goals  of  student  learning  were  then  assigned  to 
groups  of  five  people  to  consider,  discuss  and  make  recommendations.  For  some 
of  the  sessions  the  groups  were  organized  on  the  basis  of  subject  interest  and  began 
their  discussions  on  the  goals  specifically  related  to  the  subject  or  group  of  subjects. 
During  later  sessions  the  process  was  varied  by  starting  with  the  more  abstract 
goals  which  tended  to  relate  to  all  subject  areas.  Lists  of  existing  indicators  and 
measures  to  describe  them  were  accumulated  as  well  as  requirements  for  new 
measures  to  be  developed.  These  were  provided  to  the  next  group,  which 
considered  the  goal  either  at  the  beginning  of  their  discussions  or  at  the  conclusion. 
At  the  end  of  the  group  discussion,  a general  closure  session  was  conducted  to  tie 
the  process  together  and  to  encourage  on-going  discussion  and  submissions. 


In  addition  to  identifying  indicators  and  measures,  the  meetings  succeeded  in 
increasing  awareness  and  support  for  the  project  and  the  use  of  multiple  indicators 
since  there  was  a general  concern  about  using  single  indicators  to  describe  the 
quality  of  education.  They  also  served  as  a catalyst  for  participants  to  think  about 
what  schools  are  responsible  for  accomplishing.  As  a result  the  statement  of  goals 
of  student  learning  was  refined.  Suggestions  were  made  about  clusters  of 
indicators  and  interpretation  of  results,  such  as  the  relationship  between 
achievement,  participation  rates  and  dropouts,  the  grade  levels  at  which  measures 
were  suitable  and  the  nature  of  survey  instruments.  The  process  also  served  to 
make  the  participants  realize  and  accept  that  quality  indicators  are  a reality  of  life  at 
this  time  and  that  it  is  important  for  them  to  be  involved  in  the  identification  of 
suitable  measures.  The  opportunity  for  participants  to  influence  and  mold  the 
review  system  so  that  it  is  suited  the  schools  of  the  area  was  considered  extremely 
important 


After  initial  identification  of  the  indicators  and  measures  by  parents  and  teachers,  a 
full  review  by  project  personnel  was  conducted  to  ensure  consistency  from  goal  to 
goal  and  to  ensure  necessary  measures  of  performance  for  each  goal.  These 
indicators  and  measures  were  then  reviewed  by  each  of  the  local  steering 
committees  and  categorized  as  a high  or  low  priority.  The  high  priorities  were 
those  which  would  be  developed  immediately,  whereas  the  low  priorities  were 
those  that  will  be  developed  later  in  the  project 


Although  an  initial  set  of  indicators  and  measures  for  student  outcomes  had  been 
identified,  it  was  recognized  that  further  review  and  development  was  necessary  to 
ensure  that  appropriate  measures  were  identified  and  developed.  In  particular,  the 
recent  inventory  of  assessment  instruments  (Alberta  Education,  1990)  needed  to  be 
examined  to  determine  whether  there  were  suitable  instruments  for  measuring  some 
of  the  goals  of  learning.  Furthermore,  investigation  of  possible  use  of  measures 
from  other  Alberta  Educational  Quality  Indicator  projects  needed  to  be  considered, 
particularly  those  projects  involved  in  identifying  indicators  of  affective  and 
behavioral  outcomes. 
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The  Indicators,  Measures  and  Comparators : Through  the  workshop  sessions  and 
the  review  of  the  literature,  a list  of  outcome  indicators  relevant  to  any  system 
review  were  identified.  The  indicators  were  identified  in  three  categories:  cognitive, 
affective  and  behavioral.  The  cognitive  indicators  included  achievement,  program 
participation,  completion  rates,  independent  learning  and  problem  solving;  the 
affective  indicators  were  student  satisfaction,  self-concept,  self-esteem,  values, 
attitudes,  motivation  and  aspirations;  the  behavioral  indicators  were  desirable 
characteristics,  attendance  and  locus,  community  service,  image,  health,  fitness, 
participation  in  related  activities  and  post  secondary  registrations.  Appropriate 
comparators  were  also  identified  for  all  indicators.  Table  1 gives  an  example  of 
clusters  of  indicators,  the  measures  and  comparators  for  one  goal  of  learning: 
developing  knowledge,  skills  and  attitudes  in  language  and  communication. 


The  measures  to  describe  student  outcome  indicators  generally  fall  into  three  major 
categories: 

• tests  which  include  diploma  examinations,  provincial  achievement  tests, 
standardized  tests  and  locally  developed  tests; 

• existing  documents  such  as  records  and  reports;  and 

• surveys  including  opinion  questionnaires  and  participation  in  activities. 


In  general,  there  appear  to  be  sufficient  measures  that  are  readily  available  in  each  of 
the  three  areas.  The  five  exceptions  are  measures  for:  speaking  and  listening; 
writing  skills  for  grades  8 and  11;  commitment  to  the  use  of  resources  and 
preservation  of  the  environment;  fitness,  nutrition  and  hygiene  and  healthy 
lifestyles.  Additionally,  the  survey  instruments  for  obtaining  opinions  from 
students,  parents,  the  public,  employees  and  alumni  needed  to  be  refined  while  at 
the  same  time  ensuring  consistency  with  the  previously  - administered  instruments 
in  the  jurisdictions. 


Information  will  be  collected  directly  through  examinations  and  surveys;  from 
school  personnel,  eg.  attendance  and  enrollment  data,  discipline  data  and  school 
activities  that  promote  goals;  and  other  data  such  as  census  data  will  be  gathered  by 
the  project  personnel  through  document  review.  Following  development  or 
identification  of  indicators  and  measures,  it  will  be  necessary  to  fully  develop  the 
logistics  of  the  collection  procedures.  At  this  point  in  the  project,  the  context,  input 
and  process  variables  influencing  student  outcomes  have  not  been  identified.  Once 
more,  it  will  be  vital  to  have  full  involvement  of  teachers  and  parents  in  this 
process  to  build  acceptance  for  the  specific  indicators.  When  these  indicators  are 
identified,  it  will  then  be  possible  to  describe  the  analysis  of  the  collected 
information  essential  to  the  interpretation  process  which  follows. 


A Comprehensive  Process 


9 


County  of  Lacombe, 
Rocky  Mountain  School  Division 


Examples  Of  Outcomes,  Indicators,  Measures,  Comparators,  And  Grades 
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The  Interpretative  Framework,  and  Process:  The  interpretative  framework,  which  is 
proposed,  is  designed  to  facilitate  an  understanding  of  student  outcome  indicators 
on  tiie  goals  of  student  learning.  In  each  jurisdiction  and  in  each  school  the  student 
outcomes  will  be  reviewed  in  relation  to  the  identified  comparators  as  well  as 
process,  input  and  context  variables  to  determine  whether  the  outcomes  are 
appropriate.  In  this  evaluative  process  the  variables  will  also  be  identified  that  are 
contributing  to  the  outcomes. 


An  interpretative  team,  comprising  representatives  of  all  educational  partners,  will 
be  formed  for  each  goal  of  student  learning  in  each  school  jurisdiction.  After  all  of 
the  teams  have  met  to  interpret  the  outcomes  on  specific  goals,  the  steering 
committee  in  each  jurisdiction  will  review  the  results  of  the  individual  interpretation 
teams  to  identify  overall  strengths,  weaknesses  and  the  process,  input  and  context 
variables  contributing  to  the  outcomes.  Following  these  two  interpretation 
processes  each  school  will  interpret  its  student  outcomes  through  a similar  structure 
and  process. 


The  process  of  interpretation  will  consist  of  five  interrelated  steps: 

• completion  of  the  tests  and  surveys  by  the  interpretation  teams  members; 

• establishment  of  acceptable  and  desirable  levels  of  performance; 

• determination  of  the  relationship  of  the  outcomes  to  the  comparators  and  the 
interrelationship  of  the  indicators  for  each  goal; 

• determination  of  the  process,  input  and  context  variables  contributing  to  the 
outcomes; 

• identification  of  strategies  to  improve  the  outcomes. 


The  interpretation  process  for  individual  goals  will  commence  with  the  team 
members  actually  doing  each  test  and  completing  the  survey.  This  activity,  carried 
out  prior  to  the  team  meeting,  familiarizes  the  participants  with  the  test  or  survey  as 
well  as  the  level  of  difficulty  of  the  items.  Next  participants  agree  on  both  an 
acceptable  and  a desirable  level  of  performance  for  both  the  item  and  its  aggregate, 
the  outcome.  Having  the  interpreters  establish  these  levels  before  they  see  the 
actual  results  not  only  deepens  their  understanding  of  the  test  or  survey  but  it  also 
reduces  their  tendency  to  rationalize  rather  than  interpret  the  actual  results. 


As  a baseline  for  discussion,  the  initial  interpretative  activity  in  this  step  will  be  to 
determine  whether  the  outcomes  are  above  or  below  the  identified  comparator.  This 
will  be  followed  by  an  examination  of  the  interrelationship  of  the  indicators  for  each 
goal  to  determine  a tentative  level  of  performance  on  the  goal.  For  example,  the 
interpretation  of  the  student  outcomes  for  English  30  could  be  as  follows:  The 
system  results  on  the  English  30  diploma  exam  were  five  percent  above  the 
provincial  average  and  one  percent  above  the  past  district  average.  These  averages 
were  identified  as  the  comparators.  While  this  result  would  appear  to  be  very 
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positive,  the  related  indicators  would  need  to  be  examined.  On  further  examination, 
it  was  found  that  the  participation  rate  for  English  30  was  two  percent  above  the 
provincial  participation  rate  for  this  course.  Furthermore,  a review  of  the  student 
survey  results  showed  that  the  student  attitude  and  satisfaction  with  English  was 
very  good,  and  student  involvement  in  related  activities  (such  as  the  student 
newspaper)  was  high. 


At  this  point  the  interpretation  team  would  be  required  to  make  an  evaluative 
judgement  about  the  student  outcomes  on  a four-point  scale.  In  this  instance  they 
would  likely  judge  the  outcome  to  be  on  the  positive  end  of  the  scale.  On  the  other 
hand,  if  the  indicators  were  not  uniformly  positive,  the  evaluation  of  the  outcomes 
at  this  stage  could  only  be  tentative  until  a full  review  of  the  process,  input  and 
context  variables  was  completed  in  the  next  step. 


Following  the  team  review  of  the  relationship  of  the  student  outcomes  to  the 
comparators  and  the  interrelationship  of  the  indicators,  the  team  will  determine  the 
variables  that  are  contributing  to  the  positive  results  or  identify  the  variables  that  are 
inhibiting  more  positive  student  outcomes.  The  team  will  begin  by  examining  the 
variables  most  amenable  to  change,  the  instructional  processes,  to  determine  the 
processes  that  are  contributing  to  positive  outcomes  and  those  that  are  not.  In 
addition,  parental  activities  that  contribute  to  outcomes  will  be  reviewed  at  this 
point  Next,  input  variables  followed  by  context  factors  will  be  reviewed  to 
determine  the  effect  on  student  outcomes. 


At  the  conclusion  of  this  step,  the  interpretation  team  will  be  able  to  identify  the 
process,  input  and  context  variables  that  are  contributing  to  the  student  outcomes 
for  the  goal  as  well  as  those  that  are  not.  Furthermore,  the  team  will  be  able  to 
make  a final  assessment  of  student  performance  on  the  goal. 


As  an  integral  part  of  the  review  of  process,  input  and  context  variables  related  to 
student  outcomes,  the  team  will  identify  areas  requiring  improvements.  For 
example,  it  may  be  found  that  the  primary  mode  of  instruction  in  social  studies  is 
teacher  - centered  lecturing  and,  therefore,  more  extensive  utilization  of  student  - 
centered  approaches  such  as  cooperative  learning  and  peer  coaching  would  be 
recommended  to  improve  student  performance.  These  areas  suggested  for 
improvement  would  be  further  reviewed  later  as  a part  of  the  system's  overall 
strategic  planning  process. 


Following  the  interrelation  of  the  outcomes  for  each  goal,  the  system  steering 
committee  will  review  the  results  of  the  individual  interpretation  process  to  ensure 
consistency  in  interpretation  from  goal  to  goal  and  to  identify  common  strengths, 
weaknesses  and  improvement  strategies  across  the  goals.  The  process  employed 
by  the  steering  committee  will  be  a review  of  the  results  of  the  individual 
interpretation  teams  rather  than  a re-interpretation.  The  conclusions  of  the  steering 
committee  deliberations  will  be  the  substance  of  the  public  report  on  the  system 
review. 
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The  final  interpretation  activity  will  be  the  school  level  interpretation  of  student 
outcomes  in  relation  to  the  system  interpretations.  A similar  interpretation  process 
will  be  used  at  school  level  as  at  the  system  level,  that  is  , teams  for  each  goal  and 
an  overall  review  team,  both  involving  representatives  of  all  educational  partners. 
The  purpose  of  this  activity  will  be  to  identify  individual  school  strengths, 
weaknesses  and  improvement  strategies.  This  information  will  be  reported  to  the 
school’s  parents  and  school  community. 


Future  Developments,  Observations  and  Comments 

The  primary  activity  during  the  first  year  of  the  project  was  to  conceptualize  and 
develop  a full  school  system  review  process  based  on  a quality  indicator  system. 
The  intent  was  to  develop  a comprehensive  process  commencing  with  delineation  of 
goals  of  student  learning,  identification  and  development  of  indicators,  description 
of  collection,  analysis  and  interpretative  procedures  and  development  of  a strategic 
plan  to  maintain  strengths  and  improve  on  weaknesses.  All  developmental  activities 
were  to  involve  the  steering  committees  in  each  jurisdiction  as  well  as  the 
educational  partners  directly  for  components  where  enhancing  acceptance  and 
ownership  of  the  developments  was  vital. 


Developments  to  be  Completed 


A number  of  components  in  the  developmental  phase  still  require  completion.  The 
measures  for  some  student  outcome  indicators  require  further  development  unless 
existing  valid  and  reliable  instruments  can  be  identified.  As  well,  the  context,  input 
and  process  indicators  have  to  be  determined  by  direct  consultation  with  parents  and 
educators.  Once  these  developments  are  completed  the  collection  and  analysis 
procedures  for  measures  of  all  indicators  can  be  finalized.  Furthermore,  the 
reporting  procedures  and  the  follow-up  planning  process  need  to  be  developed.  All 
of  these  activities  will  occur  prior  to  field  testing  and  refinement  of  the  instruments 
in  year  two. 


Observations  and  Comments 


Those  involved  with  the  project  believe  that  the  comprehensive  school  review 
process  based  on  an  indicator  system  has  the  potential  for  positively  influencing 
student  learning  and  public  support  for  education.  The  fact  that  parents,  educators, 
support  personnel  and  trustees  have  been  significantly  involved  in  the 
developmental  phase  will  enhance  acceptance  and  ownership  and  increase  the 
likelihood  that  teaching  and  learning,  policy  directions  and  parental  and  public 
support  for  education  will  change  positively  as  a result  of  the  review. 
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The  components  of  the  project,  being  consistent  with  the  literature,  further  increase 
the  likelihood  of  acceptance  and  success.  Initiating  the  project  on  a sound  basis  of 
agreed-upon  goals  of  learning,  following  with  identification  of  clusters  of  outcome 
indicators  for  each  goal,  identifying  the  context,  input  and  process  variables, 
measuring  the  outcomes,  interpreting  the  results  and  developing  a strategic  plan  for 
improvement  is  seen  as  a fair  and  comprehensive  process.  In  particular,  it  should 
be  noted  that  educators  have  been  very  concerned  that  judgements  about  the 
performance  of  the  educational  system  are  often  made  on  the  basis  of  single 
indicators  .This  review  process  based  on  multiple  indicators  of  the  school  system 
process  will  rectify  that  concern. 


An  additional  advantage  of  the  system  review  process  is  that  it  is  "incorporated  into 
a results  chain  that  links  all  organizational  effort"  (Kaufman,  1988,  p.80).  In  fact, 
the  project  is  based  on  the  fundamental  purpose  of  school  systems,  student 
learning,  it  involves  all  partners  in  development  and  implementation,  and  it  will 
result  in  a strategic  plan  for  improvement  The  components  of  that  plan  will  be 
incorporated  into  the  job  responsibilities  of  personnel.  This  system,  therefore,  has 
the  potential  to  influence  the  actions  of  parents,  educators  and  trustees.  That 
potential  is  enhanced  because  of  the  broad  acceptance  of  the  project 


Looking  ahead:  In  further  developing  the  project,  care  needs  to  be  exercised  to 
ensure  that  the  indicator  system  is  manageable  from  the  educator's  perspective. 
While  it  is  essential  to  utilize  a comprehensive  set  of  indicators  and  measures,  there 
needs  to  be  caution  to  make  certain  that  the  number  of  indicators  is  not 
overwhelming.  Staff  has  suggested  that  distribution  of  outcome  measures 
throughout  the  grades  will  assist  in  making  the  review  process  more  manageable. 
That  is  being  addressed.  Furthermore,  while  participation  in  the  developmental 
process  is  important,  it  must  not  become  a burden  to  those  fully  engaged  in 
teaching. 


Overall,  there  is  strong  support  for  the  project  in  the  school  jurisdictions.  The 
educational  partners  have  appreciated  the  opportunity  to  mold  the  nature  of  the 
indicator  system.  They  believe  that  once  implemented,  the  system  will  provide 
two  benefits:  a comprehensive  picture  of  each  jurisdiction's  performance,  and 
provide  the  information  necessary  to  the  continual  improvement  of  student 
learning. 
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Appendix 

County  of  Lacombe  and 
Rocky  Mountain  School  Division 
Goals  of  Student  Learning 

Goal  1:  Develop  knowledge,  skills,  and  positive  attitudes  in  language  and 

communication,  mathematics,  practical  and  fine  arts,  sciences,  and  social 
studies. 

Goal  2:  Develop  different  modes  of  inquiry  and  learning  including: 

• skills  of  finding,  comprehending,  organizing,  analyzing,  and  applying 
information; 

• skills  of  learning  through  technology; 

• skill  of  studying. 

Goal  3:  Develop  intellectual  curiosity  and  a desire  for  independent  life-long  learning. 
Goal  4:  Develop  a sense  of  community  responsibility  which  embraces: 

• an  understanding  of  and  appropriate  participation  in  citizenship  at  the 
local,  national,  and  international  levels; 

• respect  for  law  and  authority,  public  and  private  property,  and  the 
opinions  and  rights  of  others; 

• appreciation  of  the  importance  of  traditional  and  culture. 

Goal  5:  Develop  a commitment  to  the  careful  use  of  natural  resources  and  to  the 
preservation  of  the  physical  environment 

Goal  6:  Acquire  knowledge  and  develop  skills  which  contribute  to  physical  well- 
being. 

Goal  7:  Acquire  knowledge,  develop  skills,  attitudes,  and  habits  that  contribute  to 
emotional  well-being  including: 

• achieving  a positive  self-concept; 

• acquiring  a high  level  of  self-discipline  and  individual  responsibility; 

• acquiring  an  ability  to  respond  to  change; 

• developing  short  and  long-term  personal  goals. 

Goal  8:  Acquire  knowledge  and  skills,  attitudes,  and  habits  for  individuals  to  be 
successful  and  respond  to  the  opportunities  and  expectations  of  the  world. 

Overall  Goal:  Achieve  successful  graduation  from  senior  high  school. 
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Paper  presented  as  part  of  the  symposium,  "Educational  Quality  Indicators: 
Collaboration  in  Action",  at  the  annual  meeting  of  the  Canadian  Educational 
Researchers’  Association,  Victoria,  June  5,  1990. 


Measuring  Social  Competence  in  Students 


The  purpose  of  this  paper  is  to  outline  a conceptualization  of  social  competence  and 
a methodology  for  measuring  social  competence.  Some  discussion  will  be 
provided  of  difficulties  in  accomplishing  the  conceptualization  and  methodology, 
as  well  as  a discussion  of  future  stages  in  the  application  of  the  conception  and 
methodology  to  one  school  jurisdiction. 

Introduction 


Background 

Fort  McMurray  Public  School  District  #2833  is  made  up  of  13  schools  - ten 
elementary/junior  schools,  two  high  schools  and  one  special  programs  school.  All 
are  located  within  the  city.  The  District's  fundamental  purposes  are  to  ensure  that 
all  4,700  students  achieve  levels  of  knowledge  and  skill  consistent  with  their 
varying  abilities;  that  they  perform  at  levels  equal  to  or  greater  than  established 
norms;  that  they  develop  positive  attitudes  towards  learning  and  toward  themselves 
and  others;  and  that  they  develop  into  productive  citizens. 

The  District  has  been  involved  in  the  development  of  Student  Performance 
Indicators  and  Standards  since  1982.  The  first  Annual  System  Student  Evaluation 
Report  (1986-1987)  was  presented  to  Trustees  by  administration  in  October  1987. 
The  report  drew  together  a variety  of  information  representing  the  educational 
health  of  students  in  the  system:  academic,  behavioral/emotional/attitudinal, 
physical,  and  cultural.  Some  of  the  systems  to  identify  and  collect  the  information 
took  almost  four  years  to  put  into  place.  The  first  report  represented  a major  effort 
to  provide  "measurable"  evidence  to  the  Trustees  and  the  public  of  the  health  of  the 
District's  educational  efforts. 

Although  the  defined  outcomes  and  standards  of  the  District's  "indicator  system" 
represented  a solid  basis  for  future  development  towards  being  increasingly 
accountable  to  the  ratepayer  for  educational  benefits,  some  indicators  and  their 
standards  were  quickly  identified  as  needing  revision  and  refinement.  The 
indicators  and  standards  requiring  most  revision  were  in  the  area  of  social  skills, 
behaviors  and  attitudes.  In  particular,  there  was  deemed  to  be  a need  to  shift  from 
"negative"  indicators  to  more  "positive"  or  desirable  indicators.  This  study  was 
designed  to  address  this  need  for  revision. 

Rationale 

First  of  all,  schools  appear  to  be  becoming  more  accountable  for  knowledge,  skill 
and  attitudes  outcomes  related  to  a variety  of  topical  social  areas,  such  as  smoking, 
environment,  A.I.D.S.,  suicide  prevention  and  so  forth.  More  broadly  conceived, 
schools  appear  to  be  becoming  increasingly  responsible  for  development  of  socially 
competent  young  people.  Thus  there  is  a practical  need  to  understand  what 
constitutes  a socially  competent  individual,  how  to  assess  the  competencies,  and 
how  to  report  on  the  selected  competencies.  Additionally,  all  of  this  needs  to  be 
done  in  ways  which  are  valid  and  reliable,  yet  feasible  in  an  operational  sense. 
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Secondly,  there  is  a need  for  further  research  into  the  general  construct  of  social 
competence  and  its  assessment.  The  existing  theories  concerning  social 
competence  and  skills  appear  to  lack  cohesiveness  and  consistency. 

Purpose  of  the  Study 

The  purpose  of  the  study  was  to  develop  a set  of  social  skills,  behaviors  and 
attitude  indicators  and  standards  which  would  then  be  used  to  assess  the  quality  of 
a selected  portion  of  one  school  jurisdiction’s  educational  program  and  delivery. 

To  fulfill  the  study  purpose,  it  is  necessary  to  complete  the  following  tasks: 

1 . to  identify  or  develop  a set  of  desirable  student  social  skills; 

2 . to  identify  or  develop  measurable  outcomes  and  standards 
for  the  desirable  student  social  skills; 

3 . to  establish  a methodology  for  collecting,  analyzing  and 
interpreting  the  data; 

4 . to  identify  or  develop  strategies  for  teaching  the  identified 
desirable  student  social  skills; 

5 . to  identify  or  develop  a means  for  reporting  information  and 
findings  to  users. 

Social  Competence 

Systems  Conceptualization 

A systems  perspective  of  social  competence  based  on  a modified  CIPP  model 
(Worthen  and  Sanders,  1987,  p.78)  was  used  as  the  conceptual  framework  to 
discuss  the  concepts  in  this  study,  and  their  interrelationships  as  conceived  and 
schematically  presented  in  Figure  1.1. 

Worthen  and  Sanders  (1987)  note  that: 

Until  we  have  solid  information  about  the  relative  effectiveness  of 
the  numerous  evaluation  approaches,  choices  among  alternatives  will 
remain  a matter  of  the  evaluator's  preference.. ..(since)  there  is 
almost  no  research  to  guide  one’s  choice,  (pp.  148-149) 

They  further  state  that: 

adherence  to  any  one  model  rather  than  another  is  largely  a 

statement  of  philosophy  or  a profession  of  faith. 
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Figure  1.  Systems  Model  of  Competent  Social  Response 


Since  no  universally  accepted  approach  to  the  conceptualization  and  evaluation  of 
social  competence  appears  to  exist,  and  since  no  universally  accepted  approach  to 
evaluation  appears  to  exist,  a decision  was  made  to  use  a systems  perspective  as  the 
basic  organizing  framework  for  the  conceptualization.  The  systems  perspective 
was  compatible  with  the  beliefs  of  the  study  team  and  appeared  to  provide  good 
information  for  program  development  and  student  evaluation  related  to  social 
competence.  The  concepts  which  were  pertinent  to  this  study  included  social 
knowledge,  skills  and  attitudes;  social  information  processing;  social  responses  or 
social  outcomes;  and  social  context. 

Conceptualization  of  the  Construct. 

The  review  of  the  literature  concerning  the  nature  of  social  skills  and  social 
competence  quickly  identified  certain  common  threads  which  were  pertinent  to  the 
conceptualization  that  evolved. 

1 . There  are  specifiable  elements/skills. 

2 . There  is  a need  for  the  elements/skills  to  be  processed  in  an 
appropriate  manner;  the  processes  may  be  either  cognitive  or 
affective. 

3 . The  appropriateness  of  behaviors  to  specific  situations  is  crucial. 

4.  There  is  no  agreed-upon  generic  social  skills  listing.  Also,  any 
list  needs  to  be  flexible  to  some  extent  to  the  needs  of  the  user. 

5 . Appropriate  social  responses  are  affected  by  the  developmental 
level  of  the  respondent,  and  the  audience  for  the  respondent 

6 . Social  responsibility  can  be  learned. 

7 . The  whole  social  competence  construct  is  dynamic,  synergistic, 
and  organic. 

8 . Specific  social  elements  and  skills  are  comprised  of  specific  and 
discrete  verbal  and  non-verbal  behaviors  and  entail  both  effective 
and  appropriate  initiations  and  responses. 

9 . The  skills  are  interactive  by  nature. 

10.  Deficits  and  excesses  in  social  response  can  be  specified  and 
targetted  for  intervention. 

1 1 . Instructors  attempting  to  teach  social  skills  or  appropriate  social 
responses  should  adhere  to  the  "relevance  of  behavior  rule" 

(Allyon  and  Azrin,  1968)  which  states  that  instructors  should 
teach  only  those  behaviors  that  will  continue  to  be  naturally 
reinforced  after  training.  Adopting  this  viewpoint  assists  in 
ensuring  that  social  skills  selected  for  instruction  will  have  some 
intrinsic  value  to  the  child,  some  benefits  for  the  child,  and  be 
valued  by  others  who  would  likely  reward  their  occurrence. 
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Social  Environment  ( Context):  The  construct  of  social  environment  evolved  to 
include  developmental  level,  audience,  and  situation.  The  "developmental  level" 
came  to  include  concepts  of  age,  language  development,  physical  development,  and 
academic  competency.  For  practical  purposes,  they  were  defined  as  Divisions  I,  n, 
ID  and  IV,  as  they  relate  to  school  systems.  "Situation"  was  further  defined  in  an 
effort  to  simplify  the  construct  and  yet  provide  some  forms  of  recognition  and 
consistency  to  it  without  making  it  totally  unmanageable.  Situations  were  identified 
as  old/new,  familiar/unfamiliar,  and  friendly/hostile.  "Audience"  came  to  be 
defined  as  those  individuals  receiving  the  responses.  The  audiences  might  include 
peers,  teachers,  parents  or  'significant  others'.  'Significant  others'  could  include 
older  or  younger  children,  or  significant  other  adults. 

Social  Information  ( Knowledge , Skills  and  Attitudes):  The  construct  of  social 
information  that  evolved  relied  heavily  on  Reschly  and  Gresham  (1981)  who 
identified  social  competence  as  being  composed  of  two  components:  adaptive 
behavior  and  social  skills.  The  social  skills  were  further  identified  as  breaking  into 
three  categories:  interpersonal  behaviors,  self-related  behaviors  and  task-related 
behaviors.  Examples  of  interpersonal  behaviors  included  accepting  authority, 
conversation  skills,  cooperative  behaviors  and  peer  relationships.  Self-related 
behaviors  involved  expressing  feelings,  ethical  behavior,  and  positive  attitude  to 
self.  Examples  of  task-related  behaviors  included  attending  behavior,  completing 
tasks  and  following  directions. 

Social  Information  Processing:  While  it  was  recognized  that  there  were  a series  of 
identifiable  skills,  these  in  turn  required  some  fonn  of  processing  and  are  related  to 
what  Reschley  and  Gresham  (1981)  refer  to  as  'adaptive  behavior'.  These  adaptive 
behaviors  include  independent  functioning  skills,  physical  development,  language 
development  and  academic  competencies.  These,  in  turn,  were  all  recognized  as 
operating  within  some  form  of  context,  and  that  context  was  defined  as  the 
developmental  level  of  the  respondent  as  well  as  the  audience  which  the  respondent 
was  referencing,  and  the  specific  situation.  Therefore  a model  of  social  competence 
evolved  which  included  a series  of  social  skill  components,  adaptive  behavior 
processes  and  contextual  variables.  This  model  was  reviewed  by  individuals  from 
Alberta  Education  and  suggestions  were  made  regarding  potential  revisions.  The 
adaptive  processes  which  had  been  identified  as  academic  competencies,  language 
development,  physical  development  and  independent  functioning  skills,  were  cross- 
referenced  to  other  aspects  of  the  model.  Language  development  and  physical 
development  were  viewed  as  sub-sets  of  the  contextual  variable  of  developmental 
level.  Academic  competency,  while  obviously  having  a relationship  to  social  skill 
development,  was  not  deemed  to  be  a crucial  focus  for  the  purposes  of  this  study, 
although  there  is  a recognition  that  there  is  a correlation.  That  left  one  key  adaptive 
behavior  process  identified  as  "independent  functioning  skills".  In  an  attempt  to 
refine  the  concept  of  independent  functioning  skills,  a further  review  of  the  literature 
was  undertaken  to  identify  and  clarify  what  independent  functioning  skills  might 
entail. 
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A selective  review  of  literature  concerning  cognition,  decision-making  and  problem- 
solving was  undertaken.  In  the  process,  an  application  by  Perry  and  Perry  (1987) 
of  Dodge's  Social  Information  Processing  Model  of  Social  Competence  was  found. 
The  model  describes  cognitive  steps  thought  necessary  to  children's  appropriate  and 
competent  action  in  social  situations,  namely: 

1 . Encoding  social  cues. 

2.  Interpreting  behavior. 

3 . Generating  alternative  responses. 

4 . Choosing  a response  after  evaluating  potential  consequences 
of  alternatives. 

5 . Performing  the  chosen  response. 

A futher  description  by  Perry  and  Perry  (1987)  notes  that  encoding  of  social  cues 
involves  searching  for  relevant  social  information  before  responding. 

Interpretation , involves  giving  meaning  to  the  cues  intended.  The  response  search, 
generates  various  possible  behavioral  responses  to  the  situation  at  hand,  and  these 
responses  can  vary  in  quantity  as  well  as  quality.  The  response  decision  involves 
choosing  a response  after  evaluating  the  potential  consequences  of  each  possible 
response.  And  the  final  step  is  enactment  where  there  is  a behavioral  performance 
of  the  chosen  response,  and  children  obviously  cannot  perform  successfully  the 
response  they  have  selected  as  best  unless  they  possess  the  motor  and  self- 
regulatory  capacities  to  carry  it  out  This  conceptualization  was  deemed  to  more 
completely  define  the  concept  of  the  "processing"  of  the  social  knowledge,  skills 
and  attitudes  which  were  deemed  pertinent  to  the  study. 

Social  Responses:  Definition  of  social  response  areas  is  difficult  since  they  need  to 
be  acceptable  as  norms  of  social  behavior  that  are  widely  accepted  across  and  within 
different  groups  in  society.  Therefore,  it  was  decided  to  adopt  the  social  outcomes 
or  norms  identified  as  desirable  within  Alberta  and  specified  by  the  province  in  the 
§chQ<?!  Aft  and  in  the  Guide  to  Education. 

The  1988  Alberta  School  Act  specifies  a code  of  conduct: 

A student  shall  conduct  himself  so  as  to  reasonably  comply  with  the 
following  code  of  conduct: 

a.  be  diligent  in  pursuing  his  studies; 

b . attend  school  regularly  and  punctually; 

c.  cooperate  fully  with  everyone  authorized  by  the 
Board  to  provide  education  programs  and 
other  services; 

d.  comply  with  the  rules  of  the  school; 

e . account  to  his  teachers  for  his  conduct; 

f . respect  the  rights  of  others,  (p.  1 1 ). 

Additionally,  some  desirable  personal  characteristics,  outlined  by  Alberta  Education 
in  A Guide  to  Education  (1989),  are  categorized  into  three  areas: 

a.  ethical/moral  characteristics; 

b.  intellectual  characteristics;  and 

c.  social/personal  characteristics. 
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The  ethical/moral  characteristics  included  respectful,  responsible,  fair/just,  tolerant, 
honest,  kind,  forgiving,  committed  to  democratic  ideals  and  loyal.  The  intellectual 
characteristics  included  open-minded,  thinks  critically,  intellectually  curious, 
creative,  pursues  excellence  and  appreciative.  The  social/personal  characteristics 
included  cooperative,  accepting,  conserving,  industrious,  possesses  a strong  sense 
of  self-worth,  persevering,  prompt,  neat,  attentive,  unselfish,  and  mentally  and 
physically  fit. 

Additionally,  John  Raven,  (1982),  indicates  that: 

....the  vast  majority  of  teachers,  pupils  and  parents  want  schools  to 
foster  such  qualities  as  the  willingness  and  the  ability  to  take  the 
initiative  in  introducing  change  into  their  society,  independence,  the 
ability  to  make  their  own  observations  and  learn  without  instruction, 
the  ability  to  apply  facts  and  techniques  to  new  problems,  to  develop 
their  characteristics  and  personality,  and  to  ensure  that  they  leave 
school  intent  on  being  master  of  their  destinies,  (p.342). 

Raven  states  that  these  opinions  are  "correct”,  as  indicated  in  a variety  of  research 
efforts  between  1961  and  1979,  which  identifies  these  factors  as  the  most  important 
qualities  for  our  pupils  to  develop  in  relation  to  their  futures  at  work  and  in  society. 
He  further  notes,  however,  that  despite  this  agreement  about  what  should  be  done, 
most  secondary  school  teachers  neglect  these  wider  goals  and  concentrate  on 
achieving  academic  goals  which  probably  represent  only  a sub-set  of  what  the 
majority  of  people  wish  to  have  happening  in  schools. 

Summary  of  the  Construct:  For  socially  appropriate  responses,  or  competent 
social  response,  students  will  be  required  to  process  social  knowledge,  skills  and 
attitudes  and  respond  in  ways  that  are  appropriate  to  a particular  social 
environment.  Wien  students  are  perceived  and  received  positively  by  the 
respondent,  then  they  will  be  viewed  as  socially  competent. 

Research  Design  and  Methodology 

The  process  of  assessment  of  social  competence  appears  to  be  highly  complex.  In 
summary,  the  review  of  the  literature  notes  that: 

1 . No  single  assessment  methodology  is  sufficient.  There  appears  to  be  a need 
for  multiple  methods  of  assessment  Michelson  et  al.  (1983)  refer  to  this  as 
a comprehensive  assessment  strategy,  and  Schwartz  and  Kaplan  (1981) 
would  refer  to  it  as  triangulation. 

2 . There  are  major  issues  of  reliability,  validity  and  practicality  in  social 
competence  assessment.  The  validity  issues  focus  on  whether  or  not  the 
child  has  only  a knowledge  of  a specific  behavior,  or  can  also  perform  the 
behavior  under  appropriate  circumstances.  Reliability  relates  to  whether  or 
not  the  various  'raters'  of  social  behavior  are  consistent  with  one  another 
(for  example,  adults  to  children,  researcher  to  parent,  parent  to  teacher,  and 
so  forth).  The  issue  of  practicality  notes  that  the  assessment  strategy  must 
have  adequate  time,  trained  personnel,  resource  materials,  and 
administrative  capabilities  in  analysis.  If  these  factors  do  not  exist, 
technical  problems,  inaccurate  or  incomplete  data  sets,  and  overall  reduction 
in  quality  of  the  evaluation  may  occur. 
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On  the  other  hand,  procedures  which  are  too  time  consuming,  and  which 
fail  to  produce  useable  information  will  result  in  failure  of  staff  to  implement 
the  strategy.  From  the  point  of  view  of  operationalizing  the  comprehensive 
assessment  strategy  in  a school  context,  the  issue  of  practicality  is 
fundamental  to  the  effective  implementation  of  the  total  process. 

3 . No  single  informant  can  provide  a valid  and  reliable  perspective.  Therefore, 
there  needs  to  be  assessment  input  from  multiple  sources. 

4 . The  purposes  of  the  assessment  need  to  be  clearly  defined  as  either 
diagnostic,intervention,  or  both. 

5 . The  most  conventional  practical  assessment  approaches  may  be  forms  of 
self-reports  and  behavior  checklists.  Self-reports  tap  into  the  child's 
knowledge  of  social  skills  and  provides  information  related  to  their 
cognitive  and  affective  understanding  of  social  skills.  However,  they  may 
not  provide  an  accurate  assessment  of  the  child’s  everyday  performance  of 
social  skills.  Behavioral  checklists  by  knowledgeable  adults  can  provide 
information  related  to  the  actual  and  observable  social  behaviors. 

6 . A potentially  useful,  but  less  conventional  method  includes  peer 
assessment,  although  this  methodology  is  cumbersome,  relatively  new  to 
behavior  assessment,  and  may  have  questionable  reliability  with  young 
children.  Nevertheless,  it  does  collect  data  from  the  most  important  member 
of  the  child's  social  environment  and  will  reflect  an  assessment  of  actual 
social  behavior  as  perceived  by  peers. 

7 . Peers,  teachers  and  parents  constitute  the  users  of  the  student's  social 
competence  and,  therefore,  are  important  data  sources  regarding  the 
student’s  social  competence. 

In  conclusion,  no  single  approach  emerges  as  the  best  method  for  social  skills 
assessment  in  all  social  contexts.  Therefore  there  is  a need  for  the  development  of 
a comprehensive  assessment  strategy  which  will  be  valid,  reliable  and  practical; 
based  on  multiple  information  sources  and  multiple  assessment  methods;  suitable 
to  a variety  of  developmental  levels,  audiences  and  situations;  and  focus  on  the 
identification  of  social  skills,  processes  and  outcomes. 

The  major  methodological  techniques  employed  in  educational  measurement  include 
questionnaires,  interviews,  observations  and  document  analysis.  In  an  effort  to 
ensure  a "comprehensive  assessment  approach",  an  attempt  is  being  made  to  use  all 
methods  in  some  fashion. 

The  methods  have  been  cross-referenced  to  potential  respondents  as  outlined  in 
Figure  2.  Figure  2 was  then  extended  to  determine  which  of  the 
method/respondents  were  most  applicable  to  the  various  sub-systems  of  the  Model 
of  Competent  Social  Response  (Figure  1).  The  results  are  presented  in  Figure  3. 
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Figure  2.  An  Assessment  Method! Respondent  Matrix 
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Figure  3.  Comprehensive  Assessment  Strategy  or  Analysis  of  Suitable  Assessment  Method 
Approaches  for  Sub-systems  of  the  Competent  Social  Response  System  Model 


Discussion 


Social  Competence 

The  study  team  consists  of  two  teachers,  two  administrators,  two  counsellors,  a 
Supervisor  of  Student  Services  and  a senior  administrator.  The  study  team  initially 
attempted  to  come  up  with  a simple  listing  of  desirable  social  skills  and  realised  that 
the  range  of  skills  was  substantial.  Secondly,  the  team  quickly  realised  that,  in 
terms  of  selecting  the  requisite  or  desired  skills,  a number  of  other  factors  came 
into  play  such  as  the  situations  in  which  skills  needed  to  be  used,  how  the  skills 
were  processed,  and  so  forth.  At  this  point,  the  team  decided  that  a 
conceptualization  of  social  competence  was  necessary  prior  to  being  able  to  identify 
contributing  skills  and  identifying  desired  outcomes. 

The  study  team,  from  a job  point-of-view,  had  varying  backgrounds.  Additionally, 
training  was  varied,  and  included  administration,  curriculum,  and  education 
psychology.  Attempts  to  conceptualize  the  area  of  social  competence  resulted  in  a 
variety  of  schematic  approaches  ranging  from  matrices  to  cubes  to  concentric 
circles.  In  each  instance,  the  individuals  proposing  their  conceptual  frameworks 
could  not  understand  why  other  individuis  on  the  team  were  having  difficulty 
accepting  the  conception.  Eventually  it  was  realised  that  each  team  member,  from 
the  point-of-view  of  their  personal  styles  of  learning  and  training,  had  a 
conceptually  different  approach  to  educational  issues  including  the  one  of  social 
competence.  The  systems  framework  which  came  to  be  used  was  the  one  model 
which  all  study  team  members  could  accept. 

At  this  point,  the  conceptualization  became  broken  into  segments.  There  was  a 
consistent  acceptance  of  the  skills  area,  or  what  has  come  to  be  known  as  the  social 
information  sub-system,  as  being  composed  of  self-related  issues,  interpersonal- 
related  issues  and  task-related  issues.  How  the  information  was  to  be  processed 
lacked  agreement  until  the  Dodge  conception  of  social  information  processing  was 
identified.  Thirdly,  there  was  a general  acceptance  that  the  definition  by  the  School 
Act  and  the  Guide  to  Education  of  socially  desirable  outcomes  would  be  suitable 
for  our  needs.  Lastly,  team  members  agreed  that  all  of  these  factors  operated  in 
some  context  that  eventually  came  to  be  defined  as  developmental  level,  situation 
and  audience. 

The  second  major  difficulty  in  terms  of  conceptualizing  social  competence  had  to  do 
with  framing  the  situational  variable.  There  is  no  practical  method  for 
accommodating  all  of  the  various  situations  in  which  students  must  operate,  and  yet 
the  situation  has  a great  deal  to  do  with  how  children  are  defined  in  terms  of  their 
social  competence.  Therefore,  a fairly  simplistic  definition  of  the  situational 
variable  as  old  or  new,  hostile  or  friendly,  came  to  be  accepted  as  a step  towards 
practical  definition  for  purposes  of  instrument  development. 

In  summary,  the  construct  of  social  competence  was  approached  by  a variety  of 
individuals  with  a variety  of  backgrounds  who  came  to  realise  that  the  area  of  study 
was  much  larger  and  much  more  complex  than  they  had  originally  envisioned. 
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Secondly,  attempting  to  understand  the  concept  and  to  define  it  came  to  be  a very 
time-consuming  and  difficult  task.  However,  through  the  process  of  grappling 
with  the  construct,  the  team  has  basically  educated  itself  as  to  the  various 
components  and  relationships,  and  has  provided  for  themselves  a solid  foundation 
for  the  remaining  developmental  work  concerning  the  construct.  The  next  stages 
for  construct  development  will  be: 

1 . an  identification  of  specific  knowledge,  skill  and  attitude  aspects  of  the 
social  information  sub-system. 

2 . refinement  of  the  social  response  outcomes. 

Methodology 

There  was  an  early  realization  by  the  study  team  that  measurement  of  social 
competence  is  extremely  difficult  and  complex.  The  measurement  would  require 
multiple  methodologies  and  multiple  respondents,  and  that  the  methodologies  and 
respondents  would  have  to  relate  to  the  various  sub-systems  of  the  conceptual 
framework.  These  features  would  be  compounded  by  the  need  for  any 
methodologies  to  be  valid,  reliable,  and  practical.  The  study  team  initially 
considered  the  development  of  its  own  instruments.  However  it  was  recognised 
that  instrument  development,  to  make  it  valid  and  reliable,  is  an  exceptionally  time- 
consuming  task  which  was  either  beyond  the  capability  or  the  desires  of  the 
working  group.  As  a result  the  committee  decided  to  re-examine,  on  an  item-by- 
item basis,  all  the  various  assessment  instruments  which  had  been  identified,  and 
which  fit  certain  criteria. 

1 . The  instruments  would  focus  on  students  in  regular  classrooms. 

2 . There  would  be  a variety  of  developmental  levels,  largely  defined  as 
Divisions  I,  n,  ID  and  fv. 

3 . Preferably,  there  would  be  a variety  of  sub-scales  and  items  which 
addressed  the  social  response  areas  identified,  and  would  have  the  potential 
for  revision  to  meet  the  requirements  of  the  team's  particular  needs. 

4 . The  instrument  would  focus  on  positive  social  behavior,  as  opposed  to 
negative  social  behavior,  or  lack  of  social  behavior. 

5 . Any  instruments  would  also  have,  hopefully,  instrument  variations  pertinent 
to  the  various  audiences  who  might  be  responding  such  as  parents,  teachers 
and  students. 

The  search  for  instruments  has  been  undertaken  but  is  not  completed  at  this  time. 
Two  instruments  which  appear  to  have  some  strong  possibilities  are  the  Self- 
Perception  Profile  for  Children  (1985)  and  the  Self -Perception  Profile  for 
Adolescents  (1988)  which  were  developed  by  Susan  Harter.  These  scales  appear 
to  fit  the  noted  selection  criteria.  Additionally,  in  conversation  with  the  author,  it 
has  been  determined  that  new  sub-scales  can  be  developed  or  that  some  of  the 
existing  sub-scales  can  be  dropped.  In  other  words,  the  profile  is  open  to 
developmental  work  to  match  more  closely  the  conception  for  the  study.  Lastly, 
the  scale  has  norms  established  for  Alberta,  and  those  norms  will  shortly  be 
reported  as  part  of  another  project  of  Alberta  Education. 
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A second  instrument  is  being  considered  at  the  moment.  The  instrument  is  the 
Teacher  Rating  of  Social  Skills  for  Children  (TROSS-C)  developed  by  Clarke, 
Gresham  and  Elliott  (1985).  The  TROSS-C  is  a checklist  format  and  would  be 
suitable,  therefore,  for  observations  concerning  actual  behavior.  The  Harter  scales 
relate  more  to  student  perception  of  behavior. 

Thirdly,  a decision  has  been  made  to  review  some  nomination  scales  which  were 
identified  in  the  original  literature  review.  The  nomination  scales  will  be  used  with 
peers  in  terms  of  peer  assessment  of  social  competence.  Therefore,  at  this  stage  of 
the  study  there  is  serious  consideration  being  given  to  questionnaire,  checklist  and 
nomination  scales,  in  terms  of  a selection  of  established  instruments. 

Additionally,  interview  schedules  for  use  by  either  the  teacher  and/or  counsellors 
are  being  considered.  Lastly,  a document  review  process  will  be  considered  as  part 
of  the  assessment  strategy.  The  documents  that  could  be  reviewed  would  include 
disciplinary  reports  or  anecdotal  teacher  records,  such  as  a log  or  journal,  which 
would  be  available  in  a semistructured  format.  Needless  to  say,  the  bulk  of  our 
work  over  the  next  few  months  will  focus  on  some  refinements  of  the  conceptual 
element  for  both  the  overall  construct  and  for  methodology. 

Once  the  team  has  identified  instruments  which  appear  to  have  substance,  and 
utility,  it  will  be  attempting  to  combine  the  various  assessment  methodologies  and 
respondents  pertinent  to  the  various  sub-systems,  and  to  develop  an  integrative 
mechanism  for  analyzing  the  data  such  that  teachers  will  be  able  to  eventually  report 
to  parents  concerning  the  social  competence  of  their  children  as  it  relates  to  the 
social  competence  objectives  of  the  school  system.  Such  a reporting  mechanism , 
while  still  needing  to  be  defined,  would  most  likely  be  represented  on  some  form  of 
report  card.  The  major  premise  behind  this  form  of  reporting  is  that,  if  we  can 
sufficiently  define  the  social  objectives  of  a school  jurisdiction,  and  if  those  social 
objectives  are  consistent  with  the  objectives  of  society,  and  if  schools  have  a 
responsibility  for  developing  these  objectives  and  outcomes  in  students,  then  as  a 
school  jurisdiction  we  should  have  identifiable  means  of  being  able  to  demonstrate 
to  society,  to  parents,  and  to  children,  that  we  have  accomplished  the  task  that  has 
been  defined  for  us  by  the  larger  community.  Obviously  it  remains  to  be  seen  as  to 
whether  or  not  we  can  do  this  in  a fashion  that  is  acceptable  to  that  larger  society, 
and  to  parents,  and  to  do  so  in  a fashion  that  is  relatively  straightforward  and 
practical  from  the  point-of-view  of  the  profession. 
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Signs  of  Learning  in  the  Affective  Domain 


"The  Alberta  community  lives  with  a conviction  that  man  is  unique  and  is  uniquely 
related  to  his  world.  Generally,  but  not  universally,  this  expresses  itself  spiritually, 
through  the  belief  in  a Supreme  Being  (e.g.  God)"  (Alberta  Education,  1990,  p.8). 

The  Lethbridge  Catholic  School  District  operates  with  the  conviction  that  providing 
excellent  academic  education  be  done  "...  in  an  environment  of  love,  self-respect 
and  respect  for  others." 

This  research  project  is  responding  to  convictions  such  as  "...affective  education  is 
a necessary  condition  for  effective  education"  (Beane,  1986,  p.27). 

In  spite  of  these  convictions,  the  importance  of  affective  education  is  not  reflected 
in  the  educational  reporting  system  with  anything  like  the  same  intensity.  Indicators 
of  affective  quality  are  not  systematically  recorded  or  reported  as  indicators  of 
success  or  achievement. 

This  paper  outlines  the  importance  of  the  affective  domain  as  stated  in  Alberta 
Education  documents.  It  describes  the  efforts  of  the  Lethbridge  project  to  identify 
attitudes,  observe  actions,  record  behaviors  and  determine  growth  in  order  to  reflect 
the  importance  of  affective  learning  in  the  reporting  system. 

A brief  background  of  the  Lethbridge  Catholic  system  shows  its  concern  not  only 
with  intellectual  growth  and  skill  mastery,  but  also  with  spiritual,  moral  and  emo- 
tional growth.  The  project  fits  well  with  the  Prayer  of  St.  Francis. 

Lord , make  me  an  instrument  of  Your  Peace 
Where  there  is  hatred , let  me  sow  love; 

Where  there  is  injury,  pardon; 

Where  there  is  doubt , faith; 

Where  there  is  despair,  hope; 

Where  there  is  darkness,  light; 

Where  there  is  sadness,  joy; 

Some  of  the  literature  which  served  as  a basis  for  the  project  is  cited.  The  rationale 
and  procedures  followed  by  the  Lethbridge  project  are  described.  The  behaviors 
which  are  proposed  as  indicators  of  affective  growth  are  divided  into  positive 
responses  that  persons  could  make  to  life,  to  self,  to  others,  to  the  world  and  to 
learning.  Since  the  project  is  still  developing  the  theoretical  framework  and  identify- 
ing behaviors  to  be  used  as  indicators,  the  observations  are  limited  and  the  implica- 
tions are  posed  as  questions. 


Introduction 

It  is  clear  from  the  literature  and  the  documents  of  Alberta  Education  and  the 
Lethbridge  Catholic  School  District  that  the  affective  domain  is  important.  This 
project  seeks  ways  of  enabling  teachers  to  express  this  importance  in  the  reporting 
system. 

Purposes 

The  School  Handbooks  for  Elementary,  Junior  High  and  Senior  High  Schools  in 
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Alberta  (Alberta  Education,  1990)  set  out  six  specific  goals  of  schooling.  In  addi- 
tion to  the  competencies,  skills  and  knowledge  goals,  the  Handbooks  also  list  as 
goals: 

- attitudes  in  mathematics,  the  practical  and  fine  arts,  the  sciences  and  the  social 
studies. 

- attitudes  and  habits  which  contribute  to  physical,  mental  and  social  well-being. 

- an  understanding  of  the  meaning,  responsibilities,  and  benefits  of  active  citi- 
zenship. 

- attitudes  and  habits  required  to  respond  to  the  opportunities  and  expectations  of  the 

world  of  work. 

One  purpose  of  this  project  is  to  find  indicators  which  will  identify  these  attitudes. 

The  School  Handbooks  also  list  the  broader  goals  of  education  which  are  to  be 
shared  by  other  agencies  within  the  community,  especially  the  family.  These  goals 
state  that  the  school  will  strive  to: 

- develop  a sense  of  responsibility. 

- develop  a positive  self-image. 

- develop  an  appreciation  for. . . , a sense  of  purpose  in. . . , an  interest  in. . . , a commit- 

ment to...,  etc. 

These  broader  goals  are  to  be  shared  with  other  agencies  in  society,  however, 

" . . .the  actions  of  teachers  and  the  activities  that  take  place  in  schools  contribute  in  a 
major  way  to  the  formation  of  attitudes."  (Alberta  Education,  1990,  p.8)  This  puts  a 
large  portion  of  the  responsibility  for  these  affective  goals  on  the  schools.  The 
formation  of  attitudes  is  an  important  ingredient  in  the  success  of  education.  A 
second  purpose  of  this  project  is  to  assist  the  schools  in  their  contribution  to  the 
formation  of  positive  attitudes  by  devising  a system  for  monitoring  and  recording 
behaviors  which  are  indicators  of  attitudes. 

The  Handbooks  for  Alberta  schools  state  that  "...parents  and  other  groups  in  society 
clearly  expect  teachers  to  encourage  the  growth  of  positive  attitudes  in  students." 
(Alberta  Education,  1990,  p.8)  A third  purpose  of  this  project  is  to  devise  a system 
for  teachers  to  determine  if  growth  in  the  affective  domain  has  taken  place  and  if 
their  efforts  at  encouraging  positive  attitudes  have  been  successful. 

The  Handbooks  also  state  that  these  attitudes  are  the  prerequisites  to  the  develop- 
ment of  essential  personal  characteristics,  of  which  26  are  listed.  This  project  aims 
to  identify  ways  in  which  the  development  of  desirable  personal  characteristics  can 
be  recorded  through  the  use  of  observable  actions  and  products. 

Background 

For  the  past  100  years,  the  Catholic  School  system  and  the  Public  School  system 
have  shared  in  the  active  and  vigorous  task  of  education  in  Lethbridge.  The  Catholic 
System  has  1 High  School,  1 Junior  High  School  and  6 Elementary  schools  with  a 
total  of  152  teachers  and  55  support  staff  to  serve  their  2700  students.  The  Catholic 
schools  integrate  all  students  and  are  committed  to  the  full  development  of  each 
student  in  their  care.  People  in  the  schools  like  to  think  of  the  schools  as  places  of 
hope. 

The  Lethbridge  Catholic  School  District  has  a history  of  concern  for  the  affective 
domain.  Their  Policy  of  Affirmation  of  Students  and  Staff  advocates  that  all 
members  of  the  School  District  acknowledge  the  need  for  affirmation  and  practice 
affirmation  at  all  times.  "Affirmation  aims  to  prevent  a person's  failures  and  fail- 
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ings  from  destroying  him. " (Appendix  A)  Their  aim  is  the  development  of  a whole- 
some and  authentic  self  image  for  each  student. 

Programs  in  the  Lethbridge  Catholic  School  District  are  evaluated  to  determine 
whether  they  provide  an  opportunity  for  students  to  develop  "a  maturing  Christian 
response  as  their  schooling  proceeds. " (Appendix  B)  These  responses  are  in  the 
affective  domain,  in  values,  attitudes  and  predispositions. 

The  evaluation  of  student  progress  in  the  Lethbridge  Catholic  School  District  is  a 
celebration  of  learning  which  provides  "...a  sense  of  hope  and  success."  (Appendix 
C)  The  teacher  develops  a trust  relationship  with  the  students  and  evaluates  them 
"...objectively,  consistently,  fairly  and  justly  with  the  goal  of  instilling  hope  in  the 
students  and  affirming  their  God-given  talents."  (Appendix  C)  The  teacher  has  the 
responsibility  to  make  recommendations  which  will  assist  the  student's  self-actuali- 
zation. When  Maslow  popularized  'self-actualization'  in  the  1950's  he  showed  how 
it  was  closely  related  to  self-concept,  and  how  it  affected  behavior.  The  evaluation 
of  student  progress  in  the  Lethbridge  Catholic  system  has  been  designed  to  encour- 
age the  growth  of  self-image,  to  assist  in  self-actualization,  and  to  affect  behavior. 

To  assist  in  this  process  of  affective  growth,  Principals  in  the  Lethbridge  Catholic 
School  District  are  commissioned  to  work  with  parents  and  teachers  to  recognize 
the  talents  of  all  children  and  to  promote  their  development.  Principals  assist  their 
students  "...to  see  themselves  in  relation  to  the  Gospel  message  of  love,  and  to 
develop  a healthy  self-concept. " (Appendix  D) 

These  policies  are  the  basis  for  the  Mission  Statement  which  declares  that:  "We 
strive  to  instill  a responsible  attitude  toward  the  world  and  its  people." 

It  is  from  this  background  that  the  Lethbridge  Catholic  School  District  has  come  to 
the  decision  to  research  signs  of  learning  in  the  affective  domain. 

Related  Literature 

In  1964,  Bloom,  Krathwohl  and  Masia  noted  the  lack  of  a systematic  effort  to  col- 
lect evidence  of  growth  in  affective  objectives.  They  suggested  that  the  intent  to 
measure  affective  growth  often  deteriorates  into  grading  what  can  explicitly  be 
evaluated,  or  measuring  only  dramatic  or  negative  developments.  They  argued  that 
attitudes  develop  slowly  and  cause  problems  for  teachers  who  need  results  recorded 
for  each  reporting  period.  They  also  queried  the  use  of  general  terms  such  as 
'...develops  an  interest  in  reading...'  The  scope  'of  interest'  could  range  from 
recognizing  that  someone  is  reading  to  a passionate  devotion  toward  reading. 

In  1971,  Bloom,  Hastings  and  Madaus  stated  that  progress  tests  and  achievement 
examinations  which  inform  students  of  their  mastery  of  a subject,  tend  to  build 
students'  confidence  and  belief  in  their  own  competence.  This  affects  the  students' 
level  of  interest  and  motivation  to  learn  more.  They  show  that  cognitive  outcomes 
and  affective  outcomes  are  closely  related.  However,  they  prefer  formative  evalua- 
tion and  feedback  to  students  on  their  progress  toward  affective  goals  over  summa- 
tive  grading  of  affective  behavior. 

Bloom,  Madaus  and  Hastings  (1981)  urged  teachers  to  write  clear  affective  as  well 
as  cognitive  objectives.  The  use  of  observable  actions  or  products  as  objectives 
makes  it  possible  to  evaluate  the  achievement  of  those  objectives.  Teachers  were 
advised  to  ensure  congruence  between  the  stated  and  the  evaluated  objectives.  They 
were  also  to  ensure  that  the  affective  objectives  were  desirable  ones  in  relation  to 
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the  philosophy  of  the  school  and  the  needs  of  the  students. 

Rubin  (1973)  claimed  that  the  curriculum  includes  affective  learning  whether  or  not 
we  wish  it.  He  stated  that  often  high  achievement  comes  at  the  cost  of  permanent 
insecurity,  anxiety  and  a defective  self-concept.  He  suggested  that  if  teachers  deal 
with  the  affect,  students'  emotional  well  being  improves  and  so  does  their  efficiency 
of  learning.  He  proposed  a balanced  curriculum  which  integrates  knowledge,  feel- 
ings and  behavior. 

Rubin  (1974)  said  that  our  attitudes  make  us  what  we  are.  Our  beliefs  or  percep- 
tions shape  our  attitudes  or  predisposition  to  behave  in  one  way.  Our  perceptions, 
attitudes,  beliefs  and  choices  are  the  cornerstones  of  affective  education.  Affective 
education  is  about  the  learner's  attitude  toward  self,  toward  life,  toward  school  and 
toward  purpose. 

Rubin  (1982)  stated  that  cognition  is  a powerful  force  in  the  shaping  of  attitudes, 
which  in  turn  determine  emotional  responses.  He  said  that  the  goal  of  affective 
education  is  to  bring  cognitive  judgment  on  antecedent  conditions  and  consequent 
behaviors.  He  believed  that  the  best  teachers  are  concerned  with  both  the  student 
and  the  subject;  and  that  they  excel  at  instructional  procedures  in  both  the  cognitive 
and  the  affective  domain. 

Beane  (1983),  Beane  (1986)  and  Beane  and  Lipka  (1984)  said  that  self  perceptions 
influence  school  achievement  but  school  achievement  influences  the  perception  of 
self.  He  contends  that  grading  systems  which  accentuate  the  positive,  give  feedback 
of  acceptance  from  significant  others,  and  enhance  the  self  as  learner  are  superior  to 
competitive  grading  systems  which  he  alleges  are  antithetical  to  personal  develop- 
ment and  an  obstacle  to  social  cooperation. 

Glickman  (1987)  asked  if  it  was  good  or  effective  schools  we  wanted.  He  noted 
how  the  unrehearsed  responses  to  that  question  were  predominately  affective  and 
differed  from  the  formal  responses  of  numerical  data. 

The  intention  of  this  research  is  to  find  what  those  unrehearsed  responses  are  in 
regard  to  expectations  of  a quality  education  in  Lethbridge.  The  literature  cited,  as 
well  as  a broad  range  of  other  literature,  serves  as  a basis  for  this  research  project. 
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Rationale 

The  Goals  of  Secondary  Education  and  the  Goals  of  Elementary  Education  as  listed 
in  the  Handbooks  of  Alberta  Education  (1990)  aim  to  assist  students  in  developing 
positive  attitudes. 

The  Policies  of  the  Lethbridge  Catholic  School  District  instruct  the  schools  to  instill 
responsible  attitudes. 

The  desire  of  the  schools  to  report  on  affective  outcomes  is  reflected  in  their  report- 
ing systems.  The  Elementary  Schools  use  anecdotal  reporting  where  affective 
outcomes  are  reported  through  phrases  such  as,  "...works  hard.."  or  "...is  a pleas- 
ure to  have  in  class.." 
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Parent-teacher-student  interviews  are  used  in  Senior  High,  Junior  High  and  Elemen- 
tary Schools  to  report  on  affective  growth. 

The  Junior  High  School  reports  affective  outcomes  by  using  computer  generated 
descriptors.  Although  some  dissatisfaction  with  this  method  has  been  voiced,  par- 
ents still  see  affective  descriptors  as  important  indicators  of  success  at  school. 

The  purpose  of  this  project  is  to  assist  the  schools  in  their  struggles  to  report  affec- 
tive growth  by  determining  specific  behaviors  which  indicate  growth;  and  by  devis- 
ing a comprehensive,  specific  and  precise  system  for  the  reporting  of  growth  in  the 
affective  domain. 

Procedures 

This  research  project  began  with  a survey  and  a study  of  the  extensive  literature  on 
the  subject  of  affective  education,  its  significance,  its  problems,  its  connection  with 
other  aspects  of  education  and  its  manifestation  in  observable  actions  and  products. 
The  literature  study  covered  a wide  span  of  years  of  research,  as  well  as  a wide 
scope  of  types  of  research  into  affective  outcomes.  It  included  a review  of  literature 
from  the  Department  of  Education,  the  Lethbridge  School  District,  Alberta  Educa- 
tion, and  the  popular  culture  which  makes  affective  outcome  into  marketable 
products. 

The  research  then  turned  to  the  stakeholders  of  education  in  Lethbridge.  Personal, 
open-ended,  yet  structured  and  focused  interviews  were  conducted  to  find  out  what 
the  stakeholders  of  education  in  Lethbridge  saw  as  important  in  education.  The 
interviews  were  designed  to  discover  indicators  of  success  which  were  in  the  affec- 
tive domain.  The  data  collected  was  descriptors  of  behavior  which  were  seen  to  be 
evidence  of  growth  and  success  at  school.  The  open-ended  interviews  allowed  for  a 
wide  variety  of  responses.  These  responses  were  recorded  in  plain  view  of  the 
respondents  who  were  asked  for  elaboration  or  clarification  when  required. 

Sampling  was  purposive  rather  than  random.  Interviews  were  conducted  with  90 
persons.  Those  interviewed  were  students  from  grades  1 to  12,  parents  from  each  of 
the  eight  schools  in  the  district,  trustees,  teachers  and  administrators  from  each 
school,  clergy,  professionals,  business  persons,  post-secondary  educators  from  the 
University  of  Lethbridge  and  the  Lethbridge  Community  College,  and  professionals 
involved  in  Correctional  work  at  such  places  as  the  Lethbridge  Correctional  Center 
and  Mental  Health  Services. 

Students  were  asked  in  clear  and  meaningful  language  to  describe  the  behavior  of 
students  who  do  well  in  school.  Adults  were  asked  to  describe  signs  of  a quality 
education.  The  results  of  the  interviews  were  studied  to  understand  what  expecta- 
tions of  quality  were  held  by  the  stakeholders  in  education  in  Lethbridge.  The  inter- 
view data  of  descriptors  of  quality  were  then  categorized  into  ten  areas  of  similari- 
ty. Those  ten  areas,  were  analyzed  and  consolidated  into  five  categories  of  behav- 
iors and  actions  which  indicated  quality. 

The  categories  of  behaviors  were  separated  into  those  outcomes  which: 

- gave  purpose  to  LIFE  in  a spiritual  sense; 

- enhanced  SELF  esteem; 

- resulted  in  better  interpersonal  relationships  with  OTHERS; 

- showed  concern  for  society  and  the  larger  WORLD; 

- were  conducive  to  more  efficient  LEARNING. 


Affective  Domain 


5 


Lethbridge  RCSD 


Categories  of  affective  outcomes  were  also  referred  to  in  Rubin's  research  (Rubin, 
1974). 

Analysis  of  the  interview  data  permitted  the  formation  of  four  similar  behaviors  in 
each  of  these  five  categories.  The  20  indicators  were  formed  into  a theoretical 
framework  of  signs  of  learning  in  the  affective  domain,  and  were  taken  to  17  more 
teachers  to  ask  for  their  perceptions.  The  insights  of  the  teachers  were  used  to  clari- 
fy and  refine  the  list  of  behaviors  and  attitudes  which  the  stakeholders  of  education 
in  Lethbridge  saw  as  indicators  of  growth  in  the  affective  domain. 

Further  analysis  of  the  list  of  behaviors  developed  from  the  interview  data  revealed 
a commonality  of  behaviors  within  each  of  the  categories.  Within  the  area  of  behav- 
iors responding  to  LIFE,  SELF,  OTHERS,  the  WORLD,  and  LEARNING  there 
were  indicators  which: 

- were  overall  assumptions  or  beliefs; 

- gave  hope  and  purpose  to  their  actions; 

- showed  signs  of  charity,  love  or  concern; 

- were  actions  based  on  commitment. 

A new  and  refined  draft  of  behaviors  which  are  seen  as  indicators  of  growth  in  the 
affective  domain,  was  then  mailed  to  40  people  for  validation.  The  validators  were 
asked  if  this  list  of  20  behaviors  were  indicators  of  growth  in  the  affective  domain; 
if  there  were  additional  indicators;  and  if  there  were  more  categories  of  behaviors. 

Affective  Behaviors 

These  behaviors  are  proposed  as  indicators  of  growth  in  the  affective  domain. 
Growth  is  demonstrated: 

I.  In  Response  to  Spiritual  Life,  when  a person: 

1.  Expresses  belief  in  a system  of  values  which  distinguishes  right  from  wrong  by 
displaying  truth,  honesty,  integrity  and  inner  peace. 

2.  Exemplifies  hope  and  purpose  in  life  by  using  prudence  and  good  judgment  in 
wise  decision  making. 

3.  Exemplifies  charity  by  showing  love,  acceptance,  tolerance  and  reconciliation  to 
others. 

4.  Displays  selflessness  through  considerate,  responsible  and  trustworthy  behavior. 
11.  In  Response  to  Self,  when  a person: 

1.  Exemplifies  confidence,  self-worth  and  trust  by  willingly  taking  risks  and  ac- 
cepting error  with  security,  ease  and  good  humor. 

2.  Assumes  responsibility  by  accepting  consequences  of  actions,  cooperating  with 
leaders  or  assuming  leadership  roles. 

3.  Accepts  that  a wide  range  of  emotions  such  as  joy,  sorrow,  frustration,  anger, 
jealousy  or  fear  are  appropriate  human  behavior  by  expressing  and  understand- 
ing emotions  with  increasing  maturity. 

4.  Develops  gifts  and  talents  by  showing  diligence  in  work,  perseverance  in  effort 
and  pride  in  accomplishments. 

in.  In  Response  to  Others,  when  a person: 

1 . Shows  respect  and  consideration  for  authority  while  understanding  the  legitima- 
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cy  of  dissent  by  choosing  appropriate  manners  to  agree  or  disagree. 

2.  Shows  appreciation  for  the  accomplishments  of  others  by  affirming  or  encourag- 
ing them. 

3.  Accepts  and  respects  the  ideas,  rights,  property  and  personhood  of  others  by 
using  charitable  words  and  actions. 

4.  Contributes  to  building  school  community  by  sharing  ideas  and  working  cooper- 
atively. 

IV.  In  Response  to  the  World,  when  a person: 

1.  Displays  active  citizenship  by  involvement  in  volunteer  endeavors. 

2.  Shows  concern  for  the  environment  by  careful  and  responsible  use  of  natural 
resources. 

3.  Shows  a concern  for  social  justice  and  equality  through  a sense  of  fair  play, 
social  action  and  giving. 

4.  Responds  to  the  expectations  of  the  world  of  work  by  being  present,  punctual, 
attentive,  prepared  and  good  humored. 

V.  In  Response  to  Learning,  when  a person: 

1.  Tikes  pleasure  in  the  search  for  truth  by  being  open  and  receptive  to  learning, 
affirming  it  and  reflecting  it  with  happiness  and  enthusiasm. 

2.  Displays  self-direction  and  self-motivation  by  initiating  activities,  extending 
knowledge,  exploring  possibilities  or  creating. 

3.  Displays  an  open  and  inquisitive  mind  by  accepting  challenges,  considering 
change,  and  attempting  various  problem-solving  techniques. 

4.  Displays  intellectual  curiosity  through  the  use  of  critical  thinking  and  informed 
questioning. 

These  indicators  are  in  the  process  of  being  made  more  precise,  specific,  and  usable 

by  identifying  actions  and  behaviors  which  are  appropriate  as  the  child  matures. 

The  behaviors  are  being  identified  in  consultation  with  teachers.  From  this  list  of 

specific  behaviors,  it  will  be  possible  to  devise  a system  for  reporting  affective 

growth. 


Discussion 


Observations 

1.  There  is  intense  interest  in  the  affective  domain.  It  is  evident  in  the  vast  scope  of 
research,  debate,  and  literature  on  the  affective  domain  and  its  pervading  influ- 
ence in  other  aspects  of  learning. 

2.  Interest  in  the  importance  of  self-image  and  attitude  to  success  in  life  comes 
through  in  the  message  of  the  popular  culture,  which  makes  the  building  of  self- 
esteem into  a marketable  product. 

3.  The  schools  in  the  Lethbridge  Catholic  School  District  show  interest  in  the 
affective  domain  by  continually  seeking  ways  of  reporting  affective  growth  to 
students  and  parents. 

4.  The  elementary  students  of  Lethbridge  Catholic  Schools  who  were  asked  to 
describe  successful  students  in  school  used  phrases  such  as  ’they  try  their  best' 
and  'they  smile  and  like  school'. 
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5.  High  School  students  who  were  asked  for  evidence  of  success  at  school  often 
cited  interpersonal  relationships  and  behavior  as  well  as  attitude  toward  school. 

6.  When  interviewed,  parents  often  mentioned  self-motivation,  self-confidence  and 
self-esteem  as  being  qualities  which  indicate  success  at  school. 

7.  Other  community  members  cited  open-mindedness,  intellectual  curiosity,  values 
and  attitudes  as  indicators  of  quality  in  education. 

8.  Notwithstanding  the  observed  importance  of  the  affective  domain,  there  is  no 
systematic  method  of  recording  and  reporting  affective  growth. 

Implications 

The  research  into  signs  of  learning  in  the  affective  domain  has  raised  many  ques- 
tions: 

1.  In  what  ways  can  the  recording  and  reporting  of  affective  growth  across  the 
grades  be  accomplished? 

2.  What  instruments  can  be  used  to  report  on  the  affective  domain? 

3.  What  type  of  inservice  will  be  necessary  so  that  teachers  will  be  able  to  adopt  a 
system  of  reporting  affective  growth. 

4.  What  manifestations  of  the  indicators  are  appropriate  as  the  child  matures? 

5.  How  will  the  reporting  of  affective  growth  modify  the  reporting  of  Grade 
Twelve  diploma  examination  results? 

6.  What  effect  will  reporting  on  affective  growth  have  on  school  programs,  plan- 
ning and  instruction? 

7.  Most  importantly,  what  effect  will  the  recording  and  reporting  of  affective 
growth  have  on  the  students? 

Conclusions 

The  enthusiasm  with  which  the  project  has  been  received  gives  hope  that  the 

development  of  instrumentation  will  be  accepted  as  a means  of  reporting  affective 

growth. 
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Appendix  A 

AFFIRMATION  OF  STUDENTS  AND  STAFF 

Lethbridge  Catholic  School  District  Policy  adopted  June  13,  1984 


1 . A statement  on  affirmation  adopted  by  the  Board  of  Trustees  draws  attention  to 
the  need  for  acknowledgement  of  the  presence  and  value  of  our  colleagues,  staff 
associates  and  students. 

2.  Affirmation  has  a special  place  in  Catholic  life.  The  great  sacrament  of  Recon- 
ciliation recognizes  our  weaknesses,  and  in  absolution  declares  people  well 
again.  Perhaps  as  a protest  against  emphasis  on  the  negative  aspects  of  behavior, 
some  Catholic  writers  claim  the  failure  to  affirm  as  "the  greatest  sin."  Everyone 
can  affirm  and  everyone  needs  affirmation.  Most  people  do  their  best  when  they 
do  the  commonplace;  if  affirmation  awaits  the  heroic  and  the  exceptional,  then 
few  will  experience  it.  Affirmation  as  practiced  in  school  acknowledges  the 
importance  and  worth  of  people. 

3.  Without  diminishing  the  need  for  honest  feedback,  and  criticism  on  how  we 
carry  out  our  tasks,  all  members  of  the  School  District  acknowledge  the  need 
for  the  act  of  affirmation  of  each  person  in  the  routine  performance  of  his  du- 
ties. 

4.  With  respect  to  the  student,  the  school  aims  to  have  each  one  graduate  with  a 
wholesome  self  image.  Affirmation  aims  to  prevent  a person's  failures  and  fail- 
ings from  destroying  him.  This  means  that  our  graduates,  and  others  who  leave 
our  schools  know  their  abilities  and  have  joy  in  them;  they  know  something  of 
their  limitations  and  can  appraise  their  significance. 

5.  The  Principal  initiates  a discussion  of  the  practice  of  affirmation  of  students  and 
staff  with  the  entire  school  staff  annually. 
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Appendix  B 

EVALUATION  OF  PROGRAMS 

Lethbridge  Catholic  School  District  Policy  adopted  June  12,  1985 


The  School  Board  may  require  evaluation  of  a program  within  its  jurisdiction  for 
whatever  reason  it  considers  appropriate  and  at  the  times  it  considers  most  useful. 


Background 

From  time  to  time,  the  School  Board,  its  administration  or  teaching  staff  may  dis- 
cern a need  to  review  a particular  program  in  the  District.  In  that  event,  the  School 
Board  calls  for  an  evaluation  of  the  program.  The  School  Board  has  an  abiding 
concern  to  ensure  that  the  students  develop  a maturing  Christian  response  as  their 
schooling  proceeds,  and  wants  to  ensure  that  the  program  provides  that  maturing, 
among  whatever  other  program  specific  objectives  belong  to  it. 

Guidelines 

1.  Program  means  a set  of  independent  activities  and  services  designed  to  achieve 
specific  organizational  goals,  policies  and  objectives.  Examples  of  programs 
are:  Mathematics  30;  Vocational  Education,  Counselling,  transportation  and 
school  modernization  plan. 

2.  The  evaluation  assesses  the  effectiveness  of  the  activities  in  the  program  against 
its  goals,  policies  and  objectives  as  specified  in  the  Program  of  Studies  and 
Curriculum  Guide.  The  evaluation  comments  upon  the  appropriateness  of  the 
goals,  policies  and  objectives.  Where  appropriate,  evaluation  will  determine  if 
the  program  affords  the  students  with  a developing  understanding  of  the  Gospel 
Message  as  it  applies  in  the  program. 

3.  The  evaluation  describes  the  program,  identifies  the  number  of  students  served, 
makes  commendations  and  recommendations  and  determines  the  costs. 

4.  The  evaluation  group  may  consist  of  personnel  from  the  School  District  staff 
along  with  consultative  personnel  from  Lethbridge  Regional  Office  of  Education 
or  other  school  jurisdictions  as  deemed  desirable. 
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Appendix  C 

EVALUATION  OF  STUDENT  PROGRESS  - THE  CELEBRATION  OF  LEARN- 
ING 

Lethbridge  Catholic  School  District  Policy  revised  June  12,  1986 


Ideals  of  hope,  affirmation,  reconciliation,  and  renewal  direct  and  guide  the  proc- 
esses of  evaluation  of  student  progress  and  the  celebration  of  learning.  The  goals 
and  objectives  of  the  Lethbridge  Catholic  School  District  No.  9 and  those  of  Alberta 
Education  provide  the  basis  for  these  processes. 


Background 

Each  child,  of  infinite  worth,  as  a child  of  God,  merits  the  best  attention  of  the 
school.  Although  teachers  have  a primary  role  in  providing  this  attention  during  the 
process  of  evaluation,  they  act  in  the  recognition  of  the  love  parents  have  for  their 
children.  The  process  of  evaluation  provides  the  students  with  a sense  of  hope  and 
success.  The  Board  of  Trustees  of  Lethbridge  Catholic  School  District  No.  9 sup- 
ports its  teachers  in  their  work  of  strengthening  the  students'  confidence  in  their 
abilities. 

Guidelines 

1.  The  school  and  its  classroom  teachers  attend  to  the  development  of  each  child's 
Christian  maturity. 

2.  In  each  school  the  primary  responsibility  for  the  process  of  evaluation  of  student 
progress  lies  with  the  classroom  teacher  who  develops  a relationship  of  trust 
with  the  student  and  parents.  The  principal  supplies  leadership. 

3.  In  evaluating  student  progress,  the  classroom  teacher  does  so  objectively,  con- 
sistently fairly  and  justly  with  the  goal  of  instilling  hope  in  the  students  and 
affirming  their  God-given  talents. 

4.  In  the  classroom  setting  the  teacher  has  the  responsibility  and  authority  to  make 
recommendations  which  seek  the  development  of  the  student's  self  actualization. 

5.  The  classroom  teacher  communicates  clearly,  precisely  and  in  confidence  such 
recommendations  to  other  teachers,  the  principal,  parents  and  students. 

6.  In  the  secondary  school,  the  teacher  informs  the  students  and  parents  of  the 
basis  of  the  evaluation  process.  In  the  elementary  schools,  the  classroom  teacher 
will  inform  the  parents,  upon  request,  of  the  basis  of  the  evaluation  process. 

7.  The  teacher  and  school  use  the  evaluation  of  student  progress  as  a means  of 
celebrating  the  achievement  of  the  students. 


Affective  Domain 


12 


Lethbridge  RCSD 


Appendix  D 

ROLE  OF  THE  PRINCIPAL 

Lethbridge  Catholic  School  District  Policy  adopted  March  28,  1984 


1.  The  Principal  leads  the  school:  whether  he  acts  or  not,  he  determines  the  school 
as  no  other  person  does. 

1.1  In  a Catholic  School  the  principal  creates  a climate  for  the  spiritual  develop- 
ment of  the  staff  and  students;  he  applies  Gospel  values  to  the  management  of 
the  school,  and  so  aims  to  assist  his  students  to  see  themselves  in  relation  to 
the  Gospel  message  of  love,  and  to  develop  a healthy  self-concept. 

1.2  The  principal  serves  as  instructional  leader  of  the  school,  he  understands  and 
accepts  the  social  role  of  the  school.  He  has  a commitment  to  personal  growth 
in  education:  he  learns  and  studies. 

1.3  The  principal  administers  the  school.  He  seeks  to  maintain  a collegial  rela- 
tionship with  other  teachers  on  his  staff.  He  works  with  other  members  of  the 
administrative  staff  on  the  implementation  of  School  Board  policies.  He 
manages  the  resources  of  the  school  and  serves  the  staff  and  students.  He 
provides  for  the  safety  and  comfort  of  staff  and  students. 

1.4  The  principal  accepts  the  parents  as  the  primary  educators  of  the  children: 
they  have  the  chief  responsibility  for  their  children's  education  and  a great 
influence  on  it. 

2.  The  Responsibilities  of  the  Role 

2. 1 The  principal  works  together  with  the  staff,  parish  priests  and  religion  con- 
sultant to  influence  the  school  with  the  Gospel  message.  He  arranges  for 
celebrations  acknowledging  the  liturgical  calendar  and  significant  events  in  the 
lives  of  his  students  and  staff.  He  seeks  to  develop  a supportive  environment 
so  the  child  develops  as  a whole  person.  He  understands  the  meaning  of 
visibility  in  the  Catholic  Community. 

2.2  The  principal  explains  the  school  to  interested  persons.  He  accounts  for  the 
achievement  of  the  students.  He  helps  teachers  interpret  the  achievement  of 
children.  He  establishes  a climate  of  openness  in  the  school  and  welcomes 
visitors  to  the  school.  He  knows  and  makes  use  of  the  resources  and  services 
of  the  community.  He  works  with  his  staff  in  the  development  of  an  appropri- 
ate array  of  co-curricular  activities. 

2.3  The  principal  supervises  instruction,  consults  with  teachers  and  other 
responsible  personnel  to  ensure  the  implementation  of  the  curriculum  as 
required  by  law,  regulation  and  School  Board  decision.  He  assists  teachers  to 
employ  new  methods  and  materials.  He  encourages  teachers  to  invent.  The 
principal  identifies  in-service  needs  of  staff  and  arranges  ways  of  meeting 
them.  He  participates  in  the  selection  of  staff.  He  assigns  teachers  to  subject 
areas  of  greatest  teaching  effectiveness. 

2.4  The  principal  seeks  to  recognize  the  talents  in  each  child  and  promotes  their 
development.  The  principal  works  with  parents  and  teachers  in  identifying  and 
developing  these  talents;  he  readily  consults  with  them  on  the  needs  and 
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opportunities  for  growth  and  development  of  the  student.  He  attempts  to  match 
children  and  teachers  based  upon  learning  and  teaching  styles. 

2.5  The  principal  enjoys  and  respects  the  collegial  relationship  he  has  with  other 
administrators  and  members  of  the  Principals'  Association. 

2.6  The  principal  inspects  the  school  plant  and  school  grounds  regularly  and 
recommends  needed  changes.  He  provides  routine  supervision  of  the  plant  and 
facilities.  He  ensures  the  procurement  of  materials  and  supplies  for  the 
school,  directs  the  preparation  of  the  annual  budget,  and  looks  to  the  ordering 
of  those  same  items  from  suppliers. 
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Lethbridge  RCSD 


Addresses 


Dr.  John  0.  Anderson 
Faculty  of  Education 
University  of  Victoria 
Box  1700 

Victoria,  British  Columbia 
V8W  2Y2 

Mr.  Bemie  Chandler 
County  of  Lacombe  No.  14 
Box  1330 
Lacombe,  Alberta 
TOC  ISO 

Mr.  Ralph  Himsl 
Lethbridge  RCSSD  No.  9 
534  - 18  Street  South 
Lethbridge,  Alberta 
T1J  3E7 

Ms.  Esther  Lambert 
Lethbridge  RCSSD  No.  9 
534  - 18  Street  South 
Lethbridge,  Alberta 
T1J  3E7 

Dr.  Nelly  McEwen 
Planning  and  Policy  Secretariat 
Alberta  Education 
11160  Jasper  Avenue 
Edmonton,  Alberta 
T5K  0L2 

Mr.  Roger  Mestinsek 

Grande  Prairie  School  District  No.  2357 

10213  - 99  Street 

Grande  Prairie,  Alberta 

T8V  2H3 

Dr.  Philip  Nagy 

Department  of  Measurement,  Evaluation, 

and  Computer  Applications 

Ontario  Institute  for  Studies  in  Education 

252  Bloor  Street  West 

Toronto,  Ontario 

M5S  1V6 

Mr.  Derek  Taylor 

Grande  Prairie  School  District  No.  2357 
10213  - 99  Street 
Grande  Prairie,  Alberta 
T8V  2H3 

Dr.  Earle  J.  Wamica 
Lethbridge  School  District  No.  51 
433  - 15  Street  South 
Lethbridge,  Alberta 
T1J  2Z5 

Mr.  David  G.  Young 

Fort  McMurray  School  District  No.  2833 

9401  Franklin  Avenue 

Fort  McMurray,  Alberta 

T9H  3Z7 

